Oct 30 23:59:55.134742 ip-10-0-141-63 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 23:59:55.134755 ip-10-0-141-63 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 23:59:55.134762 ip-10-0-141-63 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 23:59:55.135010 ip-10-0-141-63 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 00:00:05.337252 ip-10-0-141-63 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 00:00:05.337268 ip-10-0-141-63 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d944ecaeaad44af6b3d215cb56f6e833 -- Oct 31 00:02:06.273884 ip-10-0-141-63 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:02:06.624346 ip-10-0-141-63 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:02:06.624346 ip-10-0-141-63 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:02:06.624346 ip-10-0-141-63 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:02:06.624346 ip-10-0-141-63 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:02:06.629270 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629174 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:06.629270 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629265 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:06.629270 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629269 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:06.629270 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629273 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:06.629270 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629276 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:06.629270 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629279 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629282 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629285 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629288 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629291 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629293 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629296 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629299 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629301 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629304 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629306 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629309 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629311 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629314 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629317 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629319 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629321 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629324 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629332 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629334 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:06.629506 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629337 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629340 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629342 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629345 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629348 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629350 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629352 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629355 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629357 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629360 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629362 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629365 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629367 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629370 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629374 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629377 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629380 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629382 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629385 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:06.629979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629388 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629390 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629393 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629395 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629398 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629400 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629404 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629407 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629409 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629411 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629414 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629417 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629419 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629423 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629426 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629428 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629431 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629433 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629436 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629438 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:06.630929 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629441 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629443 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629446 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629449 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629452 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629455 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629458 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629461 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629463 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629466 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629469 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629471 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629474 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629477 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629479 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629481 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629484 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629487 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629489 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629500 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:06.631393 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629503 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629506 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629508 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629511 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629514 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629518 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629522 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629526 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629529 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629531 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629534 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629537 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629539 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629544 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629548 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629551 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629553 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629557 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:06.631876 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629559 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629562 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629565 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.629567 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631414 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631422 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631425 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631428 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631431 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631433 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631436 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631439 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631441 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631443 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631452 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631456 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631458 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631461 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631464 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631467 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:06.632333 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631469 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631472 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631475 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631478 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631480 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631483 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631488 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631492 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631495 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631498 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631501 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631504 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631507 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631509 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631512 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631514 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631517 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631520 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631522 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:06.632838 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631524 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631527 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631529 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631532 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631534 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631538 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631540 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631543 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631553 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631556 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631559 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631561 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631564 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631567 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631569 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631572 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631576 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631579 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631582 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631585 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:06.633289 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631604 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631609 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631612 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631615 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631617 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631620 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631623 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631625 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631627 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631630 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631633 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631635 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631637 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631640 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631643 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631646 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631648 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631650 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631653 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631655 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:06.633786 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631658 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631667 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631670 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631673 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631675 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631678 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631681 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631683 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631687 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631690 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631692 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631695 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631698 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631700 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631704 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631708 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631711 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631713 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631716 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:06.634285 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631718 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631721 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631723 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631725 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631728 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631730 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631732 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631735 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631737 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631741 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631743 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.631746 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632262 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632268 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632271 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632274 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632277 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632279 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632281 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632284 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:06.634741 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632287 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632290 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632293 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632295 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632298 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632301 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632304 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632306 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632309 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632312 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632314 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632317 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632320 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632322 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632325 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632327 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632330 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632333 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632335 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632338 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:06.635193 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632340 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632343 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632346 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632348 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632351 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632353 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632363 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632368 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632371 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632373 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632376 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632378 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632381 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632384 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632387 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632389 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632392 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632395 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632398 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:06.635666 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632400 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632403 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632405 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632408 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632410 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632413 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632415 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632418 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632420 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632423 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632425 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632428 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632430 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632433 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632435 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632438 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632440 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632443 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:06.636106 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632447 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632450 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632454 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632460 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632463 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632466 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632468 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632471 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632473 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632476 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632478 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632481 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632483 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632486 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632488 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632491 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632493 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632496 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632498 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632501 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:06.636532 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632503 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632506 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632508 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632511 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632513 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632515 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632518 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632520 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632523 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632525 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632528 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632530 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632533 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632535 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632538 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632541 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632549 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632552 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632557 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:06.637012 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632560 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.632563 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633497 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633507 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633514 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633519 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633524 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633527 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633532 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633536 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633539 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633542 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633546 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633549 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633553 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633556 2580 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633558 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633561 2580 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633564 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633567 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633573 2580 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633576 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633580 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633583 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:02:06.637444 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633615 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633625 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633628 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633631 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633635 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633638 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633642 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633647 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633651 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633654 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633659 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633663 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633666 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633669 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633672 2580 flags.go:64] FLAG: --enable-server="true" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633675 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633679 2580 flags.go:64] FLAG: --event-burst="100" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633682 2580 flags.go:64] FLAG: --event-qps="50" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633685 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633688 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633691 2580 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633695 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633698 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633701 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633704 2580 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:02:06.638010 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633707 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633710 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633713 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633716 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633718 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633721 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633724 2580 flags.go:64] FLAG: --feature-gates="" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633728 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633731 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633734 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633737 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633740 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633743 2580 flags.go:64] FLAG: --help="false" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633746 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633751 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633754 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633757 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633760 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633764 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633767 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633770 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633772 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633775 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:02:06.638679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633778 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633781 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633784 2580 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633787 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633790 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633794 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633797 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633800 2580 flags.go:64] FLAG: --lock-file="" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633803 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633806 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633809 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633815 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633817 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633820 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633823 2580 flags.go:64] FLAG: --logging-format="text" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633826 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633830 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633832 2580 flags.go:64] FLAG: --manifest-url="" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633835 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633840 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633843 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633847 2580 flags.go:64] FLAG: --max-pods="110" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633850 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633855 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633859 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:02:06.639212 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633862 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633865 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633867 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633870 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633878 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633882 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633884 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633888 2580 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633891 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633893 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633896 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633899 2580 flags.go:64] FLAG: --port="10250" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633902 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633905 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-073c3d8149f0f39eb" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633908 2580 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633911 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633914 2580 flags.go:64] FLAG: --register-node="true" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633917 2580 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633921 2580 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633923 2580 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633926 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633929 2580 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633933 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633936 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633939 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633942 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:02:06.639823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633945 2580 flags.go:64] FLAG: --runonce="false" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633948 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633950 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633953 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633958 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633960 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633964 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633967 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633970 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633973 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633976 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633979 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633981 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633984 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633987 2580 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633990 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633995 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.633998 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634001 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634006 2580 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634009 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634012 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634015 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634018 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634020 2580 flags.go:64] FLAG: --v="2" Oct 31 00:02:06.640479 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634025 2580 flags.go:64] FLAG: --version="false" Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634029 2580 flags.go:64] FLAG: --vmodule="" Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634033 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634036 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634192 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634197 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634200 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634204 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634206 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634209 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634211 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634215 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634219 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634221 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634227 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634230 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634233 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634235 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634238 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634240 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:06.641572 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634243 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634245 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634248 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634250 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634253 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634255 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634258 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634261 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634264 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634266 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634269 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634271 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634276 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634280 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634283 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634286 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634289 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634292 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634295 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:06.642582 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634300 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634302 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634305 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634307 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634311 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634314 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634316 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634319 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634322 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634325 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634327 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634330 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634332 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634335 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634337 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634340 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634342 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634345 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634347 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634350 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:06.643512 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634352 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634355 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634357 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634359 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634362 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634365 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634368 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634370 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634373 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634375 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634378 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634380 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634384 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634386 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634389 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634393 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634397 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634400 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634403 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:06.644388 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634405 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634408 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634411 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634414 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634416 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634419 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634421 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634424 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634426 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634429 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634431 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634434 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634437 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634439 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634442 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634444 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634447 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634449 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634451 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634454 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:06.645284 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634457 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634459 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634462 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634464 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634467 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634471 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634473 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634476 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634478 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634482 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634485 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634487 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.634514 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634551 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634555 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634559 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634562 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:06.645879 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634565 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634567 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634570 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634572 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634575 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634578 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634580 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634583 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634585 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634607 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634610 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634613 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634616 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634618 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634621 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634624 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634626 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634629 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634631 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634634 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:06.646417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634637 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634641 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634644 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634646 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634650 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634653 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634656 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634658 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634661 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634664 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634668 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634672 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634676 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634678 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634681 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634683 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634686 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634689 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634691 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:06.647245 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634694 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634696 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634712 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634716 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634719 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634722 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634725 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634728 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634732 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634735 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634737 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634740 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634742 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634745 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634749 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634752 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634755 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634758 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634760 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634763 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634766 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:06.647904 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634768 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634771 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634773 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634776 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634779 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634782 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634784 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634787 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634789 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634792 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634794 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634797 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634799 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634802 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634805 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634808 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634810 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634813 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634815 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:06.648504 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634818 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634820 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634823 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634825 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634828 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634830 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634834 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634836 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634839 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634842 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634844 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634847 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634849 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634852 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634854 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634858 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634863 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634865 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634869 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:06.649043 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634872 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634874 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634877 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.634879 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.643301 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.643323 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.643343 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.643350 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643443 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643451 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643456 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643460 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643465 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643469 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643473 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643477 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643482 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643486 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643491 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:06.649523 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643495 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643499 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643503 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643508 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643512 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643516 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643519 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643524 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643530 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643537 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643542 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643546 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643551 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643555 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643560 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643564 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643569 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643573 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643580 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:06.650025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643585 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643609 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643614 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643618 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643622 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643627 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643631 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643635 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643640 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643644 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643648 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643652 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643656 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643660 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643664 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643669 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643673 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643677 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643681 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:06.650560 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643685 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643689 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643693 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643698 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643702 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643706 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643710 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643714 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643718 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643722 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643726 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643730 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643735 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643741 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643746 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643750 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643754 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643759 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643762 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643766 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:06.651097 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643771 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643775 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643779 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643784 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643788 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643792 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643796 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643800 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643804 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643808 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643812 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643817 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643824 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643830 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643835 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643840 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643846 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643850 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643854 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643858 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:06.651892 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643863 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643867 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643871 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643875 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643880 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643885 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643890 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643894 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643898 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643903 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643907 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643911 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643915 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643919 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643924 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643928 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:06.652417 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.643932 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.643974 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644211 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644219 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644224 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644229 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644234 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644239 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644243 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644247 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644251 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644255 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644259 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644264 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644268 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644272 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:06.652979 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644276 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644280 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644284 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644289 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644293 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644297 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644302 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644307 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644312 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644338 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644346 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644352 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644370 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644376 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644380 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644385 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644390 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644395 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644399 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644403 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:06.653447 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644406 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644410 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644415 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644421 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644429 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644433 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644439 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644444 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644449 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644454 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644458 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644462 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644469 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644474 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644479 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644484 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644488 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644493 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644497 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:06.653986 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644501 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644506 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644511 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644515 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644520 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644525 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644529 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644534 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644538 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644542 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644546 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644550 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644555 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644559 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644563 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644568 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644572 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644576 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644580 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:06.654450 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644584 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644606 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644611 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644615 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644619 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644623 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644628 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644632 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644636 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644640 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644645 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644649 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644654 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644658 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644662 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644668 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644672 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644676 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644681 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644685 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:06.655025 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644689 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644694 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644698 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644702 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644706 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644711 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644715 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644720 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644724 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644728 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644732 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644736 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644740 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644745 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.644787 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644838 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:06.655726 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644845 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644850 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644854 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644860 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644864 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644869 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644873 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644877 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644882 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644886 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644890 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644894 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644899 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644904 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644910 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644914 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644919 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644923 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644928 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:06.656317 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644932 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644936 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644940 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644944 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644948 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644952 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644956 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644961 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644965 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644969 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644973 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644977 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644981 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644985 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644990 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644994 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.644999 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645003 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645007 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645012 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:06.656856 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645016 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645020 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645024 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645028 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645032 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645037 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645042 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645046 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645050 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645055 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645059 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645063 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645067 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645071 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645076 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645080 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645085 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645089 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645093 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645097 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:06.657327 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645101 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645105 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645110 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645114 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645118 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645122 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645127 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645133 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645139 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645147 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645151 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645156 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645160 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645164 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645168 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645173 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645179 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645185 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645190 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:06.657812 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645196 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645200 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645205 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645209 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645213 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645217 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645222 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645226 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645230 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645234 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645239 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645243 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645247 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645252 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645256 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645262 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645266 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645270 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645274 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:06.658259 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645279 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645282 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645286 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645292 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645296 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645300 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645303 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:06.645308 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.645390 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.648408 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.649489 2580 server.go:1014] "Starting client certificate rotation" Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.649616 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.650379 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.651917 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:02:06.658765 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.658124 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:02:06.671546 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.671528 2580 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:02:06.675220 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.675206 2580 log.go:25] "Validated CRI v1 image API" Oct 31 00:02:06.676069 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.676044 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:02:06.678618 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.678573 2580 fs.go:135] Filesystem UUIDs: map[1dbcff07-4343-445b-85a1-922f36acb372:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 84003b1f-ab97-4dbe-b9c6-4f080557f62d:/dev/nvme0n1p3] Oct 31 00:02:06.678708 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.678617 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:02:06.684110 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.683988 2580 manager.go:217] Machine: {Timestamp:2026-10-31 00:02:06.68268925 +0000 UTC m=+0.307393920 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116712 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22060b8e021cc96e8eb9a647c85114 SystemUUID:ec22060b-8e02-1cc9-6e8e-b9a647c85114 BootID:d944ecae-aad4-4af6-b3d2-15cb56f6e833 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:ad:7b:de:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:ad:7b:de:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:8f:63:eb:31:a5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:02:06.684110 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.684107 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:02:06.684267 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.684220 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:02:06.685493 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.685465 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:02:06.685671 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.685494 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-63.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:02:06.685787 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.685675 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:02:06.685787 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.685688 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:02:06.685787 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.685754 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:02:06.685787 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.685777 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:02:06.685946 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.685872 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:02:06.689496 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.689481 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:02:06.689650 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.689639 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:02:06.690486 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.690475 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:02:06.690542 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.690493 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:02:06.690542 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.690508 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:02:06.690542 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.690521 2580 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:02:06.690542 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.690532 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:02:06.690884 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.690870 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:02:06.693265 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.693232 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:02:06.699211 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.699198 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:02:06.700078 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700065 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:02:06.700111 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700096 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:02:06.700260 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700250 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:02:06.700290 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700267 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:02:06.700290 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700273 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:02:06.700290 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700278 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:02:06.700290 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700284 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:02:06.700290 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700291 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:02:06.700967 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700958 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:02:06.701001 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700968 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:02:06.701001 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700975 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:02:06.701001 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700982 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:02:06.701001 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700990 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:02:06.701001 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.700998 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:02:06.702514 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.702503 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:02:06.702554 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.702518 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:02:06.706433 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.706419 2580 server.go:1263] "Started kubelet" Oct 31 00:02:06.706547 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.706519 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:02:06.706616 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.706509 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:02:06.706616 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.706607 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:02:06.707235 ip-10-0-141-63 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:02:06.710202 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.710185 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:02:06.712096 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.712079 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:02:06.714630 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.714611 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:02:06.719980 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.719869 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:02:06.719980 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.719920 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:02:06.720120 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.719999 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-63.us-west-1.compute.internal\" not found" Oct 31 00:02:06.720739 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.720703 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:02:06.720823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.720742 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:02:06.720922 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.720766 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:02:06.721008 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.720995 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:02:06.721065 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.721010 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:02:06.721223 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.721206 2580 factory.go:55] Registering systemd factory Oct 31 00:02:06.721284 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.721226 2580 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:02:06.721452 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.721424 2580 factory.go:153] Registering CRI-O factory Oct 31 00:02:06.721452 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.721438 2580 factory.go:223] Registration of the crio container factory successfully Oct 31 00:02:06.721560 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.721490 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:02:06.721560 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.721509 2580 factory.go:103] Registering Raw factory Oct 31 00:02:06.721560 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.721527 2580 manager.go:1199] Started watching for new ooms in manager Oct 31 00:02:06.722832 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.722730 2580 manager.go:319] Starting recovery of all containers Oct 31 00:02:06.733184 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.733166 2580 manager.go:324] Recovery completed Oct 31 00:02:06.737566 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.737552 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:06.739215 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.739191 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:06.739282 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.739236 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:06.739282 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.739247 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:06.739859 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.739844 2580 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:02:06.739859 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.739858 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:02:06.739968 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.739874 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:06.739968 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.739881 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:02:06.740552 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.740536 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:06.740642 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.740570 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:06.740642 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.740584 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:06.741208 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.741191 2580 policy_none.go:50] "Start" Oct 31 00:02:06.741208 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.741206 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:02:06.741276 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.741225 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:02:06.742159 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.742149 2580 policy_none.go:44] "Start" Oct 31 00:02:06.788320 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.788181 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.793145 2580 manager.go:341] "Starting Device Plugin manager" Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.793171 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.793182 2580 server.go:87] "Starting device plugin registration server" Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.793379 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.793388 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.793482 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.793560 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.793568 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.794077 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:02:06.797903 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.794123 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-63.us-west-1.compute.internal\" not found" Oct 31 00:02:06.802413 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.802384 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:02:06.803800 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.803780 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:02:06.803887 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.803804 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:02:06.803887 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.803827 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:02:06.803887 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.803862 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:02:06.854118 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.854091 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-63.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:02:06.854118 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.854088 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-63.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:02:06.855173 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.854096 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-63.us-west-1.compute.internal.18e374476ac51672 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-63.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-63.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:02:06.706390642 +0000 UTC m=+0.331095292,LastTimestamp:2026-10-31 00:02:06.706390642 +0000 UTC m=+0.331095292,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-63.us-west-1.compute.internal,}" Oct 31 00:02:06.894220 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.894164 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:06.895029 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.895011 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:06.895137 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.895041 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:06.895137 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.895052 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:06.895137 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.895076 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:06.904600 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.904574 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal"] Oct 31 00:02:06.904700 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.904658 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:06.905382 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.905368 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:06.905470 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.905402 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:06.905470 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.905417 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:06.906680 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.906665 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:06.906829 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.906815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:06.906876 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.906845 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:06.907441 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.907425 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:06.907508 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.907465 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:06.907508 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.907475 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:06.907619 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.907427 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:06.907619 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.907547 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:06.907619 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.907562 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:06.908583 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.908566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:06.908677 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.908617 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:06.909310 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.909289 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:06.909384 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.909320 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:06.909384 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.909334 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:06.922554 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.922529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:06.922658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.922560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:06.922658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.922577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/020b2f1cd0c673bc8a3307af2e07b1c3-config\") pod \"kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"020b2f1cd0c673bc8a3307af2e07b1c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:06.933921 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.933903 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-63.us-west-1.compute.internal\" not found" node="ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:06.934056 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.934040 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:02:06.934115 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:06.934096 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rspx7" Oct 31 00:02:06.937123 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:06.937107 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-63.us-west-1.compute.internal\" not found" node="ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.023056 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.023031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.023056 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.023057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.023222 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.023076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/020b2f1cd0c673bc8a3307af2e07b1c3-config\") pod \"kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"020b2f1cd0c673bc8a3307af2e07b1c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.023222 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.023136 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/020b2f1cd0c673bc8a3307af2e07b1c3-config\") pod \"kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"020b2f1cd0c673bc8a3307af2e07b1c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.023222 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.023135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.023222 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.023137 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.111286 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.111251 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:07.111943 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.111930 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:07.112994 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.112962 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:07.113283 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.113267 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:07.113863 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.113849 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.113905 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:07.113871 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-63.us-west-1.compute.internal\": node \"ip-10-0-141-63.us-west-1.compute.internal\" not found" Oct 31 00:02:07.220892 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.220825 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.238162 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.238141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.240657 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.240642 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.348431 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.348406 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:02:07.348586 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.348516 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:07.409145 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.409110 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:02:07.649715 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.649642 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:02:07.650185 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.649809 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:02:07.650185 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.649844 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:02:07.650185 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:07.649781 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aec41e53b5c24408ea18b6975af3dda9-0bfa0ebd3c963d0d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-63.us-west-1.compute.internal.18e374476cba808d\": read tcp 10.0.141.63:45056->3.133.20.147:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-63.us-west-1.compute.internal.18e374476cba808d default 4770 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-63.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-63.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-63.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:02:06 +0000 UTC,LastTimestamp:2026-10-31 00:02:06.895055475 +0000 UTC m=+0.519760126,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-63.us-west-1.compute.internal,}" Oct 31 00:02:07.650185 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.649845 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:02:07.691174 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.691153 2580 apiserver.go:52] "Watching apiserver" Oct 31 00:02:07.720929 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.720905 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:02:07.938644 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.938516 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:57:07 +0000 UTC" deadline="2028-05-16 10:35:31.687797704 +0000 UTC" Oct 31 00:02:07.938644 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.938571 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13522h33m23.749230226s" Oct 31 00:02:07.964817 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.964792 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:02:07.965108 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.965088 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-24nkz","kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-fzxls","openshift-multus/multus-8rpzc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-mgbdr","openshift-network-operator/iptables-alerter-9c62n","kube-system/konnectivity-agent-5hrcq","openshift-dns/node-resolver-9h7rh","openshift-image-registry/node-ca-r5nkn","openshift-multus/multus-additional-cni-plugins-s4t9z","openshift-network-diagnostics/network-check-target-qgstv","openshift-ovn-kubernetes/ovnkube-node-4b5st"] Oct 31 00:02:07.966445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.966430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:07.967512 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.967482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:07.967642 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:07.967551 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:07.968565 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.968546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:07.969577 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.969559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:07.971242 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.971225 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:07.972252 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.972237 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:07.973280 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.973264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:07.973364 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:07.973311 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:07.974469 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.974449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:07.975559 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.975544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:07.976609 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.976579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:07.976696 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:07.976671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:07.978112 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.978089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:07.979225 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:07.979211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.022992 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.022627 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:02:08.022992 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.022881 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zgjmv\"" Oct 31 00:02:08.022992 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.022960 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:02:08.022992 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.022989 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:08.023227 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.023090 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:02:08.023277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.023237 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:08.023323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.023290 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.023466 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-27h5h\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.023626 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.023710 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.023803 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.023974 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5ddpr\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.024052 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.024105 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.024251 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.024452 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wr2gb\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.024793 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:02:08.025374 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.024878 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:02:08.029610 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-agent-certs\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:08.029684 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:08.029725 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-netd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.029725 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstjm\" (UniqueName: \"kubernetes.io/projected/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-kube-api-access-vstjm\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.029795 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-etc-selinux\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.029795 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-konnectivity-ca\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:08.029885 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36d4a105-7994-478f-96dc-5e168df0203c-hosts-file\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.029885 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-kubernetes\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.029885 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-host\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.029970 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.029970 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-cnibin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.029970 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flcqq\" (UniqueName: \"kubernetes.io/projected/1a785fc6-0a05-45d9-875a-6e831047d381-kube-api-access-flcqq\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030091 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.029981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-systemd-units\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030091 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-slash\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030091 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030091 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-multus-daemon-config\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030091 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-etc-kubernetes\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030251 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-node-log\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030251 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-log-socket\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030251 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-modprobe-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.030251 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/af8c82f1-cb80-4dae-a124-df184eab0f73-kube-api-access-7ktj4\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:08.030251 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-cnibin\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.030251 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030232 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-k8s-cni-cncf-io\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030440 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030440 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-multus\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030440 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-hostroot\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030440 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-script-lib\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030440 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.030440 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrbn\" (UniqueName: \"kubernetes.io/projected/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kube-api-access-zlrbn\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.030440 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdx2b\" (UniqueName: \"kubernetes.io/projected/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-kube-api-access-qdx2b\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.030440 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-cni-binary-copy\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030440 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-conf-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-kubelet-config\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-host-slash\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-system-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-var-lib-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-bin\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovn-node-metrics-cert\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-sys-fs\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030652 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ec94e45-f0e1-4e95-be72-9201b65344a5-serviceca\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-tuned\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxqw\" (UniqueName: \"kubernetes.io/projected/3b6a63f0-3097-4425-b952-e569b827a1d9-kube-api-access-tqxqw\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-netns\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-bin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.030803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ec94e45-f0e1-4e95-be72-9201b65344a5-host\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-dbus\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snd7s\" (UniqueName: \"kubernetes.io/projected/36d4a105-7994-478f-96dc-5e168df0203c-kube-api-access-snd7s\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-systemd\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-ovn\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-socket-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-conf\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.030975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-netns\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-systemd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031043 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d4a105-7994-478f-96dc-5e168df0203c-tmp-dir\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031058 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-config\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-sys\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-var-lib-kubelet\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:08.031314 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysconfig\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-system-cni-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-device-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmg5n\" (UniqueName: \"kubernetes.io/projected/4ec94e45-f0e1-4e95-be72-9201b65344a5-kube-api-access-rmg5n\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-run\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031307 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-kubelet\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-iptables-alerter-script\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-lib-modules\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-multus-certs\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-env-overrides\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvjl\" (UniqueName: \"kubernetes.io/projected/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-kube-api-access-zgvjl\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-tmp\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.031918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031580 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-socket-dir-parent\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.032343 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-kubelet\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.032343 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-etc-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.032343 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.031668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-registration-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.056652 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.056629 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gvgd9\"" Oct 31 00:02:08.068803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.068785 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7xfz6\"" Oct 31 00:02:08.068881 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.068845 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:02:08.071611 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.071581 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:02:08.072456 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072443 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqqqs\"" Oct 31 00:02:08.072515 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072465 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jt67z\"" Oct 31 00:02:08.072515 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072500 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fsk29\"" Oct 31 00:02:08.072636 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072562 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:02:08.072636 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072567 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:02:08.072740 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072662 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:02:08.072740 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072736 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:08.072810 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072762 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:02:08.072810 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072777 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:02:08.072810 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072786 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:02:08.072810 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072790 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:02:08.072964 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072949 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:02:08.072995 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.072955 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:08.073086 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.073072 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:02:08.073128 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.073088 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:02:08.074175 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.074161 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:02:08.074224 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.074209 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-njkvw" Oct 31 00:02:08.132109 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:08.132205 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysconfig\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.132205 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-system-cni-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.132205 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-device-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.132205 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmg5n\" (UniqueName: \"kubernetes.io/projected/4ec94e45-f0e1-4e95-be72-9201b65344a5-kube-api-access-rmg5n\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132207 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysconfig\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132221 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-system-cni-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-run\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132266 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-device-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-kubelet\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-iptables-alerter-script\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-lib-modules\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-multus-certs\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132376 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-kubelet\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.132410 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132406 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-run\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132451 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-env-overrides\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132507 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-lib-modules\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.132517 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132551 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132466 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-multus-certs\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvjl\" (UniqueName: \"kubernetes.io/projected/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-kube-api-access-zgvjl\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.132631 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret podName:c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:08.632573618 +0000 UTC m=+2.257278276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret") pod "global-pull-secret-syncer-24nkz" (UID: "c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-tmp\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-socket-dir-parent\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-kubelet\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-etc-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-registration-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132787 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-socket-dir-parent\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.132999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-agent-certs\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132811 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-etc-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-netd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132855 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-registration-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstjm\" (UniqueName: \"kubernetes.io/projected/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-kube-api-access-vstjm\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-etc-selinux\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132934 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-konnectivity-ca\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132923 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.132959 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132962 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-iptables-alerter-script\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36d4a105-7994-478f-96dc-5e168df0203c-hosts-file\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132942 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-env-overrides\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.132987 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-netd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.133003 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:08.632988366 +0000 UTC m=+2.257693006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-kubernetes\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.133658 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-host\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133105 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-etc-selinux\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133111 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-kubernetes\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133157 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-host\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-cnibin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133199 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36d4a105-7994-478f-96dc-5e168df0203c-hosts-file\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133212 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-kubelet\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133275 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-cnibin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcqq\" (UniqueName: \"kubernetes.io/projected/1a785fc6-0a05-45d9-875a-6e831047d381-kube-api-access-flcqq\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-systemd-units\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-slash\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133892 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-slash\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133918 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-systemd-units\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.133937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-multus-daemon-config\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-etc-kubernetes\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134152 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-konnectivity-ca\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:08.134323 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-node-log\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134009 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-log-socket\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134212 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-node-log\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-modprobe-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134241 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-etc-kubernetes\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/af8c82f1-cb80-4dae-a124-df184eab0f73-kube-api-access-7ktj4\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134269 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-log-socket\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-cnibin\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-k8s-cni-cncf-io\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-multus\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-hostroot\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-script-lib\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134398 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134406 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrbn\" (UniqueName: \"kubernetes.io/projected/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kube-api-access-zlrbn\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.135163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdx2b\" (UniqueName: \"kubernetes.io/projected/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-kube-api-access-qdx2b\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-cni-binary-copy\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134481 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-conf-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-kubelet-config\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-host-slash\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134576 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-modprobe-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-system-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-var-lib-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-bin\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovn-node-metrics-cert\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-sys-fs\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134753 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-conf-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134758 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-multus-daemon-config\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ec94e45-f0e1-4e95-be72-9201b65344a5-serviceca\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134782 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-sys-fs\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-tuned\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.136007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134832 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-cnibin\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134836 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134843 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-host-slash\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxqw\" (UniqueName: \"kubernetes.io/projected/3b6a63f0-3097-4425-b952-e569b827a1d9-kube-api-access-tqxqw\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134887 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-var-lib-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134904 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-kubelet-config\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-netns\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134924 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-bin\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-bin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-system-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ec94e45-f0e1-4e95-be72-9201b65344a5-host\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.134998 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-k8s-cni-cncf-io\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-dbus\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snd7s\" (UniqueName: \"kubernetes.io/projected/36d4a105-7994-478f-96dc-5e168df0203c-kube-api-access-snd7s\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135054 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-bin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-systemd\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135092 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-netns\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-ovn\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.136774 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135124 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ec94e45-f0e1-4e95-be72-9201b65344a5-host\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-socket-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135134 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-ovn\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-conf\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135182 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-multus\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-netns\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135222 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-socket-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-systemd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135226 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-hostroot\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d4a105-7994-478f-96dc-5e168df0203c-tmp-dir\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135287 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-config\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135460 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-sys\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-var-lib-kubelet\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.137277 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135622 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-var-lib-kubelet\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135719 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-script-lib\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135795 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-config\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135815 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-systemd\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135858 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-dbus\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135867 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-systemd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135897 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-netns\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.135981 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-conf\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.136015 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-sys\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.136130 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d4a105-7994-478f-96dc-5e168df0203c-tmp-dir\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.136206 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-cni-binary-copy\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.136215 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ec94e45-f0e1-4e95-be72-9201b65344a5-serviceca\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.136734 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-tmp\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.136742 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.136932 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-agent-certs\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.137269 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovn-node-metrics-cert\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.137871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.137822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-tuned\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.193822 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.193746 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:08.193822 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.193772 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:08.193822 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.193783 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpsr4 for pod openshift-network-diagnostics/network-check-target-qgstv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:08.194165 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.193849 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4 podName:1650ce51-15b5-4454-9c29-2a05cb986307 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:08.693825137 +0000 UTC m=+2.318529795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dpsr4" (UniqueName: "kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4") pod "network-check-target-qgstv" (UID: "1650ce51-15b5-4454-9c29-2a05cb986307") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:08.195472 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.195443 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcqq\" (UniqueName: \"kubernetes.io/projected/1a785fc6-0a05-45d9-875a-6e831047d381-kube-api-access-flcqq\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.195965 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.195947 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvjl\" (UniqueName: \"kubernetes.io/projected/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-kube-api-access-zgvjl\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.196565 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.196544 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstjm\" (UniqueName: \"kubernetes.io/projected/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-kube-api-access-vstjm\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.197312 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.197292 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmg5n\" (UniqueName: \"kubernetes.io/projected/4ec94e45-f0e1-4e95-be72-9201b65344a5-kube-api-access-rmg5n\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.200738 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.200716 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/af8c82f1-cb80-4dae-a124-df184eab0f73-kube-api-access-7ktj4\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:08.201472 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.201452 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrbn\" (UniqueName: \"kubernetes.io/projected/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kube-api-access-zlrbn\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.201958 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.201939 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdx2b\" (UniqueName: \"kubernetes.io/projected/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-kube-api-access-qdx2b\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.202662 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.202644 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxqw\" (UniqueName: \"kubernetes.io/projected/3b6a63f0-3097-4425-b952-e569b827a1d9-kube-api-access-tqxqw\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.205838 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.205816 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snd7s\" (UniqueName: \"kubernetes.io/projected/36d4a105-7994-478f-96dc-5e168df0203c-kube-api-access-snd7s\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.281213 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.281181 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f30427e3252533f78f5cc535c34236.slice/crio-ebd1860f367302fd08345faa048cd8261e0cae3ed328f609693d2f483234bb39 WatchSource:0}: Error finding container ebd1860f367302fd08345faa048cd8261e0cae3ed328f609693d2f483234bb39: Status 404 returned error can't find the container with id ebd1860f367302fd08345faa048cd8261e0cae3ed328f609693d2f483234bb39 Oct 31 00:02:08.281771 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.281746 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020b2f1cd0c673bc8a3307af2e07b1c3.slice/crio-ffff463688490cac01318c689b70f40d4c6c716da109739c5bc77448e9c49513 WatchSource:0}: Error finding container ffff463688490cac01318c689b70f40d4c6c716da109739c5bc77448e9c49513: Status 404 returned error can't find the container with id ffff463688490cac01318c689b70f40d4c6c716da109739c5bc77448e9c49513 Oct 31 00:02:08.287856 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.287553 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:02:08.300570 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.300552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:08.307670 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.307650 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d4a105_7994_478f_96dc_5e168df0203c.slice/crio-ad542f300ec38d49009840ee0f96f7ef5490fda07a8ce28005661e5b2c138c42 WatchSource:0}: Error finding container ad542f300ec38d49009840ee0f96f7ef5490fda07a8ce28005661e5b2c138c42: Status 404 returned error can't find the container with id ad542f300ec38d49009840ee0f96f7ef5490fda07a8ce28005661e5b2c138c42 Oct 31 00:02:08.314871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.314855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:08.321642 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.321620 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7be51f6_c5b1_4a67_b598_a915f5ff76e8.slice/crio-2d96e68b0eed65d882cde807a8f3b472691effd9853f4709890290672369c45a WatchSource:0}: Error finding container 2d96e68b0eed65d882cde807a8f3b472691effd9853f4709890290672369c45a: Status 404 returned error can't find the container with id 2d96e68b0eed65d882cde807a8f3b472691effd9853f4709890290672369c45a Oct 31 00:02:08.340034 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.340006 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:08.346820 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.346803 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d78a36_3b13_449d_bfb9_b25cb3e9d12d.slice/crio-77a3dc24e15b775de804b4b437886ed33b860829ce9add749e3e731d894c95db WatchSource:0}: Error finding container 77a3dc24e15b775de804b4b437886ed33b860829ce9add749e3e731d894c95db: Status 404 returned error can't find the container with id 77a3dc24e15b775de804b4b437886ed33b860829ce9add749e3e731d894c95db Oct 31 00:02:08.348656 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.348643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:08.355888 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.355862 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bedb70d_0e4f_43e3_98a7_533396bcdf84.slice/crio-2a85d6880acdb0b7fabed86323ac23bb722dd3d6c043659c960da9d2139d1150 WatchSource:0}: Error finding container 2a85d6880acdb0b7fabed86323ac23bb722dd3d6c043659c960da9d2139d1150: Status 404 returned error can't find the container with id 2a85d6880acdb0b7fabed86323ac23bb722dd3d6c043659c960da9d2139d1150 Oct 31 00:02:08.358902 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.358884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:08.365913 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.365896 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fefd28c_b0d1_4863_819d_2d273ea5e9ef.slice/crio-2db816b4acebb162a8a2eebf1591563559e709a1062f95082897f0b8b1f4a8ec WatchSource:0}: Error finding container 2db816b4acebb162a8a2eebf1591563559e709a1062f95082897f0b8b1f4a8ec: Status 404 returned error can't find the container with id 2db816b4acebb162a8a2eebf1591563559e709a1062f95082897f0b8b1f4a8ec Oct 31 00:02:08.383887 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.383870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:08.384952 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.384938 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:08.391844 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.391824 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6a63f0_3097_4425_b952_e569b827a1d9.slice/crio-a9cb1e54b0f9b26921211ce2f1ccd1697d7c67dd02a5093387d157afa0ecceb5 WatchSource:0}: Error finding container a9cb1e54b0f9b26921211ce2f1ccd1697d7c67dd02a5093387d157afa0ecceb5: Status 404 returned error can't find the container with id a9cb1e54b0f9b26921211ce2f1ccd1697d7c67dd02a5093387d157afa0ecceb5 Oct 31 00:02:08.394479 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.394459 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec94e45_f0e1_4e95_be72_9201b65344a5.slice/crio-3c34b9f4ef3558669226ac92ee841da74687c3dd89abe649c9fae0de2b2194ff WatchSource:0}: Error finding container 3c34b9f4ef3558669226ac92ee841da74687c3dd89abe649c9fae0de2b2194ff: Status 404 returned error can't find the container with id 3c34b9f4ef3558669226ac92ee841da74687c3dd89abe649c9fae0de2b2194ff Oct 31 00:02:08.412154 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.412138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:08.416238 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.416212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8rpzc" Oct 31 00:02:08.419315 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.419293 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0793ea2b_51e4_48e8_98b3_7cbbec1c410c.slice/crio-e22c37fc3709bb955af3ba23729184d35fa89c2d8191ae02bd41036659bdfe3e WatchSource:0}: Error finding container e22c37fc3709bb955af3ba23729184d35fa89c2d8191ae02bd41036659bdfe3e: Status 404 returned error can't find the container with id e22c37fc3709bb955af3ba23729184d35fa89c2d8191ae02bd41036659bdfe3e Oct 31 00:02:08.423577 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:08.423560 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a785fc6_0a05_45d9_875a_6e831047d381.slice/crio-d998baa9860603a93cbd28e6ef782132cbd53a8de34be248f98d3edd3e3e3f92 WatchSource:0}: Error finding container d998baa9860603a93cbd28e6ef782132cbd53a8de34be248f98d3edd3e3e3f92: Status 404 returned error can't find the container with id d998baa9860603a93cbd28e6ef782132cbd53a8de34be248f98d3edd3e3e3f92 Oct 31 00:02:08.425516 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.425500 2580 kuberuntime_manager.go:1707] "Unhandled Error" err="container kube-multus start failed in pod multus-8rpzc_openshift-multus(1a785fc6-0a05-45d9-875a-6e831047d381): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 31 00:02:08.425569 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.425528 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-8rpzc" podUID="1a785fc6-0a05-45d9-875a-6e831047d381" Oct 31 00:02:08.638575 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.638490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:08.638575 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.638528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:08.638774 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.638646 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:08.638774 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.638651 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:08.638774 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.638696 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:09.638684436 +0000 UTC m=+3.263389074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:08.638774 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.638708 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret podName:c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:09.638702678 +0000 UTC m=+3.263407317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret") pod "global-pull-secret-syncer-24nkz" (UID: "c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:08.682517 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.682490 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:08.739070 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.739034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:08.739197 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.739179 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:08.739197 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.739198 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:08.739289 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.739207 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpsr4 for pod openshift-network-diagnostics/network-check-target-qgstv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:08.739289 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.739259 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4 podName:1650ce51-15b5-4454-9c29-2a05cb986307 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:09.739243892 +0000 UTC m=+3.363948547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpsr4" (UniqueName: "kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4") pod "network-check-target-qgstv" (UID: "1650ce51-15b5-4454-9c29-2a05cb986307") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:08.808243 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.808173 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" event={"ID":"9bedb70d-0e4f-43e3-98a7-533396bcdf84","Type":"ContainerStarted","Data":"2a85d6880acdb0b7fabed86323ac23bb722dd3d6c043659c960da9d2139d1150"} Oct 31 00:02:08.809147 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.809121 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fzxls" event={"ID":"b7be51f6-c5b1-4a67-b598-a915f5ff76e8","Type":"ContainerStarted","Data":"2d96e68b0eed65d882cde807a8f3b472691effd9853f4709890290672369c45a"} Oct 31 00:02:08.810200 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.810177 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9h7rh" event={"ID":"36d4a105-7994-478f-96dc-5e168df0203c","Type":"ContainerStarted","Data":"ad542f300ec38d49009840ee0f96f7ef5490fda07a8ce28005661e5b2c138c42"} Oct 31 00:02:08.811260 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.811239 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"020b2f1cd0c673bc8a3307af2e07b1c3","Type":"ContainerStarted","Data":"ffff463688490cac01318c689b70f40d4c6c716da109739c5bc77448e9c49513"} Oct 31 00:02:08.812260 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.812235 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"53f30427e3252533f78f5cc535c34236","Type":"ContainerStarted","Data":"ebd1860f367302fd08345faa048cd8261e0cae3ed328f609693d2f483234bb39"} Oct 31 00:02:08.813429 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.813407 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8rpzc" event={"ID":"1a785fc6-0a05-45d9-875a-6e831047d381","Type":"ContainerStarted","Data":"d998baa9860603a93cbd28e6ef782132cbd53a8de34be248f98d3edd3e3e3f92"} Oct 31 00:02:08.814613 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.814580 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5nkn" event={"ID":"4ec94e45-f0e1-4e95-be72-9201b65344a5","Type":"ContainerStarted","Data":"3c34b9f4ef3558669226ac92ee841da74687c3dd89abe649c9fae0de2b2194ff"} Oct 31 00:02:08.815034 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:08.815014 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b14b49cabfdc258931fa03823e067322606fe50d514622d8c4f0efaadb17d1f0\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-8rpzc" podUID="1a785fc6-0a05-45d9-875a-6e831047d381" Oct 31 00:02:08.815684 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.815666 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" event={"ID":"3b6a63f0-3097-4425-b952-e569b827a1d9","Type":"ContainerStarted","Data":"a9cb1e54b0f9b26921211ce2f1ccd1697d7c67dd02a5093387d157afa0ecceb5"} Oct 31 00:02:08.816680 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.816661 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5hrcq" event={"ID":"0fefd28c-b0d1-4863-819d-2d273ea5e9ef","Type":"ContainerStarted","Data":"2db816b4acebb162a8a2eebf1591563559e709a1062f95082897f0b8b1f4a8ec"} Oct 31 00:02:08.817716 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.817696 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" event={"ID":"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d","Type":"ContainerStarted","Data":"77a3dc24e15b775de804b4b437886ed33b860829ce9add749e3e731d894c95db"} Oct 31 00:02:08.818784 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.818764 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9c62n" event={"ID":"0793ea2b-51e4-48e8-98b3-7cbbec1c410c","Type":"ContainerStarted","Data":"e22c37fc3709bb955af3ba23729184d35fa89c2d8191ae02bd41036659bdfe3e"} Oct 31 00:02:08.889153 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:08.889090 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:09.075823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:09.075776 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:57:08 +0000 UTC" deadline="2028-08-09 18:09:18.009554582 +0000 UTC" Oct 31 00:02:09.075823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:09.075817 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15570h7m8.933740029s" Oct 31 00:02:09.272065 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:09.271971 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:09.647369 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:09.647279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:09.647369 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:09.647344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:09.647619 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.647517 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:09.647619 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.647580 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:11.647559781 +0000 UTC m=+5.272264426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:09.647802 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.647684 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:09.647802 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.647721 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret podName:c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:11.647709018 +0000 UTC m=+5.272413664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret") pod "global-pull-secret-syncer-24nkz" (UID: "c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:09.748747 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:09.748708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:09.749292 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.748924 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:09.749292 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.748946 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:09.749292 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.748960 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpsr4 for pod openshift-network-diagnostics/network-check-target-qgstv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:09.749292 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.749022 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4 podName:1650ce51-15b5-4454-9c29-2a05cb986307 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:11.749001517 +0000 UTC m=+5.373706177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpsr4" (UniqueName: "kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4") pod "network-check-target-qgstv" (UID: "1650ce51-15b5-4454-9c29-2a05cb986307") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:09.804871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:09.804813 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:09.805069 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.805032 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:09.805169 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:09.805154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:09.805290 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.805243 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:09.805351 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:09.805315 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:09.805424 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:09.805400 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:10.076302 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:10.076210 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:57:08 +0000 UTC" deadline="2028-06-18 16:16:02.710285114 +0000 UTC" Oct 31 00:02:10.076302 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:10.076252 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14320h13m52.634038696s" Oct 31 00:02:11.672691 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:11.672166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:11.672691 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:11.672233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:11.672691 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.672433 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:11.672691 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.672504 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:15.672485109 +0000 UTC m=+9.297189755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:11.672691 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.672592 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:11.672691 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.672633 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret podName:c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:15.672620344 +0000 UTC m=+9.297324988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret") pod "global-pull-secret-syncer-24nkz" (UID: "c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:11.772823 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:11.772787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:11.773121 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.773100 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:11.773212 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.773128 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:11.773212 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.773144 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpsr4 for pod openshift-network-diagnostics/network-check-target-qgstv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:11.773212 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.773204 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4 podName:1650ce51-15b5-4454-9c29-2a05cb986307 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:15.773185517 +0000 UTC m=+9.397890172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpsr4" (UniqueName: "kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4") pod "network-check-target-qgstv" (UID: "1650ce51-15b5-4454-9c29-2a05cb986307") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:11.805691 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:11.804672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:11.805691 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.804809 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:11.805691 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:11.804881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:11.805691 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.804971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:11.805691 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:11.805018 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:11.805691 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:11.805095 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:12.849032 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:12.848788 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" event={"ID":"3b6a63f0-3097-4425-b952-e569b827a1d9","Type":"ContainerStarted","Data":"bdc221653c5ce31e84a06b05b20ded22d2d8d2ce854b6cd731742363dd864c7a"} Oct 31 00:02:13.804816 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.804217 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:13.804816 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:13.804391 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:13.804816 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.804478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:13.804816 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:13.804581 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:13.804816 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.804666 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:13.804816 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:13.804742 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:13.858798 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.858661 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"020b2f1cd0c673bc8a3307af2e07b1c3","Type":"ContainerStarted","Data":"9cc3a7ae20bbcf7d225c38514448b9d81aea3691e513b7d703555f5bb4319298"} Oct 31 00:02:13.861548 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.861514 2580 generic.go:356] "Generic (PLEG): container finished" podID="53f30427e3252533f78f5cc535c34236" containerID="58925b14a9fc2d5d46859193802a9073b261eb6cfa1bf19f9741a005431c1d58" exitCode=0 Oct 31 00:02:13.861714 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.861612 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"53f30427e3252533f78f5cc535c34236","Type":"ContainerDied","Data":"58925b14a9fc2d5d46859193802a9073b261eb6cfa1bf19f9741a005431c1d58"} Oct 31 00:02:13.864243 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.864177 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="bdc221653c5ce31e84a06b05b20ded22d2d8d2ce854b6cd731742363dd864c7a" exitCode=0 Oct 31 00:02:13.864352 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.864251 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" event={"ID":"3b6a63f0-3097-4425-b952-e569b827a1d9","Type":"ContainerDied","Data":"bdc221653c5ce31e84a06b05b20ded22d2d8d2ce854b6cd731742363dd864c7a"} Oct 31 00:02:13.870734 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.870106 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5hrcq" event={"ID":"0fefd28c-b0d1-4863-819d-2d273ea5e9ef","Type":"ContainerStarted","Data":"323eb13037f5ff2151e3a1673849f44af10260de552422d92d90cc3317bb438c"} Oct 31 00:02:13.986632 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:13.986579 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" podStartSLOduration=6.9865645149999995 podStartE2EDuration="6.986564515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:13.98626536 +0000 UTC m=+7.610970023" watchObservedRunningTime="2026-10-31 00:02:13.986564515 +0000 UTC m=+7.611269180" Oct 31 00:02:14.246839 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:14.246704 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5hrcq" podStartSLOduration=2.212289753 podStartE2EDuration="7.2466603s" totalImagesPullingTime="5.034370547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:14.113422215 +0000 UTC m=+7.738126884" watchObservedRunningTime="2026-10-31 00:02:14.2466603 +0000 UTC m=+7.871364966" Oct 31 00:02:14.875388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:14.875291 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r5nkn" event={"ID":"4ec94e45-f0e1-4e95-be72-9201b65344a5","Type":"ContainerStarted","Data":"6fd53572a840c7f29502e4fc4e2c23335635edf1d1bf1bb1bf5f8d0d33b761fb"} Oct 31 00:02:14.879672 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:14.879632 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"53f30427e3252533f78f5cc535c34236","Type":"ContainerStarted","Data":"1d72c23711f29fba923302406a4cc9fb4ce216f3b438c8ac00502e7d49a6e419"} Oct 31 00:02:15.130257 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:15.130159 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r5nkn" podStartSLOduration=1.132942124 podStartE2EDuration="7.130136195s" totalImagesPullingTime="5.997194071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:15.0032002 +0000 UTC m=+8.627904863" watchObservedRunningTime="2026-10-31 00:02:15.130136195 +0000 UTC m=+8.754840863" Oct 31 00:02:15.708300 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:15.708216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:15.708300 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:15.708275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:15.708530 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.708412 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:15.708530 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.708471 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:23.708454567 +0000 UTC m=+17.333159221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:15.708937 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.708907 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:15.709051 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.708977 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret podName:c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:23.708960558 +0000 UTC m=+17.333665212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret") pod "global-pull-secret-syncer-24nkz" (UID: "c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:15.805486 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:15.804923 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:15.805486 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.805140 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:15.805486 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:15.805233 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:15.805486 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.805311 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:15.805486 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:15.805370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:15.805486 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.805443 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:15.809814 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:15.809412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:15.809814 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.809667 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:15.809814 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.809693 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:15.809814 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.809707 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpsr4 for pod openshift-network-diagnostics/network-check-target-qgstv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:15.809814 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:15.809763 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4 podName:1650ce51-15b5-4454-9c29-2a05cb986307 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:23.809746085 +0000 UTC m=+17.434450725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpsr4" (UniqueName: "kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4") pod "network-check-target-qgstv" (UID: "1650ce51-15b5-4454-9c29-2a05cb986307") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:15.884492 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:15.883989 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" event={"ID":"9bedb70d-0e4f-43e3-98a7-533396bcdf84","Type":"ContainerStarted","Data":"9c13f3c1382988720021dd0a677eef4de811b12a7a9f0913c4471fa7d58a1e74"} Oct 31 00:02:17.320133 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:17.319998 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:02:17.804521 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:17.804425 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:02:17.320020648Z","UUID":"4558a21f-0213-414f-adcb-788fdd285349","Handler":null,"Name":"","Endpoint":""} Oct 31 00:02:17.805556 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:17.805077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:17.805556 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:17.805196 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:17.805556 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:17.805205 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:17.805556 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:17.805290 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:17.805556 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:17.805307 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:17.805556 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:17.805369 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:17.806128 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:17.806106 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:02:17.806250 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:17.806135 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:02:17.889007 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:17.888971 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9h7rh" event={"ID":"36d4a105-7994-478f-96dc-5e168df0203c","Type":"ContainerStarted","Data":"f0782b3ec4580a92ee98ba3257332dc03f1a74074d45b02bc93378d7dec31fb5"} Oct 31 00:02:17.891352 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:17.891270 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" event={"ID":"9bedb70d-0e4f-43e3-98a7-533396bcdf84","Type":"ContainerStarted","Data":"d034d3d53fb7dd7c4c8900e7853ff50d938b89ab99ff365f9c81c579cb6a49c7"} Oct 31 00:02:18.006062 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:18.005996 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" podStartSLOduration=11.005961019 podStartE2EDuration="11.005961019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:15.130804085 +0000 UTC m=+8.755508752" watchObservedRunningTime="2026-10-31 00:02:18.005961019 +0000 UTC m=+11.630665680" Oct 31 00:02:18.311960 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:18.311928 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:18.313057 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:18.313033 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:18.430270 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:18.430175 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9h7rh" podStartSLOduration=2.808952126 podStartE2EDuration="11.43015985s" totalImagesPullingTime="8.621207724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:18.007203502 +0000 UTC m=+11.631908164" watchObservedRunningTime="2026-10-31 00:02:18.43015985 +0000 UTC m=+12.054864510" Oct 31 00:02:18.896138 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:18.896097 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" event={"ID":"9bedb70d-0e4f-43e3-98a7-533396bcdf84","Type":"ContainerStarted","Data":"e0268cd21e5ce1051fca04aa118dad9ff3a5aa7a7c76661884310a4341d9e2b1"} Oct 31 00:02:18.899145 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:18.898646 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="37249db7afaa496b8f1769c827d3f40f1a0189309e5530ae24d08934c347e763" exitCode=0 Oct 31 00:02:18.899145 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:18.898778 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" event={"ID":"3b6a63f0-3097-4425-b952-e569b827a1d9","Type":"ContainerDied","Data":"37249db7afaa496b8f1769c827d3f40f1a0189309e5530ae24d08934c347e763"} Oct 31 00:02:19.804824 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:19.804558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:19.805253 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:19.804981 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:19.805253 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:19.804754 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:19.805253 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:19.804742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:19.805253 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:19.805067 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:19.805253 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:19.805146 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:19.903694 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:19.903440 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9c62n" event={"ID":"0793ea2b-51e4-48e8-98b3-7cbbec1c410c","Type":"ContainerStarted","Data":"4568d744a8f5afa11a121aa6ae06d83b4d6d95dff2d21c825669d8a056677649"} Oct 31 00:02:19.906262 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:19.906220 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fzxls" event={"ID":"b7be51f6-c5b1-4a67-b598-a915f5ff76e8","Type":"ContainerStarted","Data":"a9232cb870315dba81c77d01610ca0480f99aa68fd6122723cf653d6a9f58f8a"} Oct 31 00:02:20.139648 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:20.139151 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fzxls" podStartSLOduration=2.592665256 podStartE2EDuration="13.139135225s" totalImagesPullingTime="10.546469969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:20.13887574 +0000 UTC m=+13.763580401" watchObservedRunningTime="2026-10-31 00:02:20.139135225 +0000 UTC m=+13.763839885" Oct 31 00:02:20.139648 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:20.139466 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9c62n" podStartSLOduration=3.384435529 podStartE2EDuration="12.139455313s" totalImagesPullingTime="8.755019784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:20.020339297 +0000 UTC m=+13.645043975" watchObservedRunningTime="2026-10-31 00:02:20.139455313 +0000 UTC m=+13.764159976" Oct 31 00:02:20.258717 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:20.258660 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" podStartSLOduration=3.296834707 podStartE2EDuration="13.258645981s" totalImagesPullingTime="9.961811274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:20.258581987 +0000 UTC m=+13.883286687" watchObservedRunningTime="2026-10-31 00:02:20.258645981 +0000 UTC m=+13.883350635" Oct 31 00:02:20.910656 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:20.910624 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="ab70a876d71a54c2254f97dcfc78e8bf2cf031960898504dbff826346d5fa932" exitCode=0 Oct 31 00:02:20.911226 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:20.911015 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" event={"ID":"3b6a63f0-3097-4425-b952-e569b827a1d9","Type":"ContainerDied","Data":"ab70a876d71a54c2254f97dcfc78e8bf2cf031960898504dbff826346d5fa932"} Oct 31 00:02:21.804532 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:21.804499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:21.804532 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:21.804518 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:21.804816 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:21.804540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:21.804816 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:21.804658 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:21.804816 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:21.804715 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:21.804956 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:21.804824 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:22.918043 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:22.917964 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="a734dde24fde72b75da059f3f90bb1350691cb056e9cf60a96972f900a00b1f8" exitCode=0 Oct 31 00:02:22.918043 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:22.918004 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" event={"ID":"3b6a63f0-3097-4425-b952-e569b827a1d9","Type":"ContainerDied","Data":"a734dde24fde72b75da059f3f90bb1350691cb056e9cf60a96972f900a00b1f8"} Oct 31 00:02:23.774451 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.773242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:23.774451 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.773444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:23.774451 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.773561 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:23.774451 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.773625 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:39.773612326 +0000 UTC m=+33.398316977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:02:23.774451 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.774119 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:23.774451 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.774150 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret podName:c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:39.774140735 +0000 UTC m=+33.398845387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret") pod "global-pull-secret-syncer-24nkz" (UID: "c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:02:23.804336 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.804131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:23.804336 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.804248 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:23.804336 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.804310 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:23.804522 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.804391 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:23.804522 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.804318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:23.804522 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.804466 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:23.874133 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.874110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:23.874337 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.874323 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:02:23.874412 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.874345 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:02:23.874412 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.874357 2580 projected.go:196] Error preparing data for projected volume kube-api-access-dpsr4 for pod openshift-network-diagnostics/network-check-target-qgstv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:23.874500 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:23.874413 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4 podName:1650ce51-15b5-4454-9c29-2a05cb986307 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:39.874396476 +0000 UTC m=+33.499101128 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpsr4" (UniqueName: "kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4") pod "network-check-target-qgstv" (UID: "1650ce51-15b5-4454-9c29-2a05cb986307") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:02:23.926762 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.926729 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" event={"ID":"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d","Type":"ContainerStarted","Data":"0a3e30b37391beb44a9896845da96eedf7d508ca14d77b30ec09dbdb2dc8d5a1"} Oct 31 00:02:23.927202 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.926824 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" event={"ID":"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d","Type":"ContainerStarted","Data":"d9d09cf4570b9339f42c6d136b05d05a8228f14f19bf01ad1c18f3e37861bbfc"} Oct 31 00:02:23.927202 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.926875 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" event={"ID":"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d","Type":"ContainerStarted","Data":"e1df5249e57e72d6c8a6ac7a8ce6acc0b08ca288163847c1f41cef2c94385b85"} Oct 31 00:02:23.927202 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:23.926913 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" event={"ID":"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d","Type":"ContainerStarted","Data":"c4091789bcd1a8e0a7248122722083cc18c3f33e245665aab9a4b20b7d24195e"} Oct 31 00:02:24.930426 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:24.930264 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8rpzc" event={"ID":"1a785fc6-0a05-45d9-875a-6e831047d381","Type":"ContainerStarted","Data":"8b913fd43d2ef11bbada6c072b12add2393d59113a917c01817a8023e02c2101"} Oct 31 00:02:24.934833 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:24.934790 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" event={"ID":"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d","Type":"ContainerStarted","Data":"0962cf86642429d671724e2c8836c312554a3a22c436db8a91ff64e1cd895763"} Oct 31 00:02:24.934944 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:24.934871 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" event={"ID":"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d","Type":"ContainerStarted","Data":"96d9f2ca73c9ebc9256d3c2e5d2ca9266c4ff2568ed66518eb3c5a92d52e4790"} Oct 31 00:02:25.051464 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:25.051421 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8rpzc" podStartSLOduration=1.287991289 podStartE2EDuration="17.051409358s" totalImagesPullingTime="15.763418069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-10-31 00:02:25.051004852 +0000 UTC m=+18.675709512" watchObservedRunningTime="2026-10-31 00:02:25.051409358 +0000 UTC m=+18.676114018" Oct 31 00:02:25.804855 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:25.804825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:25.804855 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:25.804840 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:25.805096 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:25.804825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:25.805096 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:25.804929 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:25.805096 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:25.805021 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:25.805213 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:25.805104 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:25.942604 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:25.942555 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="4a4900f4f31f42165ceaed42fd981aacfd992d2ea8b5e1c100417f8a1d33b845" exitCode=0 Oct 31 00:02:25.943006 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:25.942631 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" event={"ID":"3b6a63f0-3097-4425-b952-e569b827a1d9","Type":"ContainerDied","Data":"4a4900f4f31f42165ceaed42fd981aacfd992d2ea8b5e1c100417f8a1d33b845"} Oct 31 00:02:26.947753 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:26.947712 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" event={"ID":"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d","Type":"ContainerStarted","Data":"388f8360a3ceea3b6015e9561db9b548cf9aebf107422845591918c3fbaef4e3"} Oct 31 00:02:26.950505 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:26.950485 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="8a2c22bda96c5ac654d80f5625ee74518e1a0a14ce316b382455b494dd656916" exitCode=0 Oct 31 00:02:26.950570 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:26.950518 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" event={"ID":"3b6a63f0-3097-4425-b952-e569b827a1d9","Type":"ContainerDied","Data":"8a2c22bda96c5ac654d80f5625ee74518e1a0a14ce316b382455b494dd656916"} Oct 31 00:02:27.804554 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:27.804516 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:27.804694 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:27.804574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:27.804694 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:27.804675 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:27.804772 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:27.804685 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:27.804772 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:27.804750 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:27.804840 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:27.804793 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:27.955729 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:27.955695 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" event={"ID":"3b6a63f0-3097-4425-b952-e569b827a1d9","Type":"ContainerStarted","Data":"102f7048adbec05117de60b89c505fb13fe78fea97c07fd48931a13256d26696"} Oct 31 00:02:28.108963 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:28.108871 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s4t9z" podStartSLOduration=9.005724462 podStartE2EDuration="21.108859448s" totalImagesPullingTime="12.103134986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:28.108363993 +0000 UTC m=+21.733068652" watchObservedRunningTime="2026-10-31 00:02:28.108859448 +0000 UTC m=+21.733564107" Oct 31 00:02:28.499023 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:28.498832 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:28.499191 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:28.499130 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:02:28.499390 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:28.499372 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:28.961745 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:28.961702 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" event={"ID":"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d","Type":"ContainerStarted","Data":"720b55edb5f85702b4a7342a69b495a6e1e47ed9ce337abd466bd81c4ab981d8"} Oct 31 00:02:29.170385 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:29.170333 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" podStartSLOduration=6.872663177 podStartE2EDuration="22.170319233s" totalImagesPullingTime="15.297656056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:29.143118662 +0000 UTC m=+22.767823321" watchObservedRunningTime="2026-10-31 00:02:29.170319233 +0000 UTC m=+22.795023892" Oct 31 00:02:29.805036 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:29.805004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:29.805278 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:29.805001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:29.805278 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:29.805104 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:29.805278 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:29.805019 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:29.805278 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:29.805173 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:29.805404 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:29.805275 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:29.964457 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:29.964428 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:29.964917 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:29.964507 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:29.964917 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:29.964545 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:29.981487 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:29.981460 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:29.981702 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:29.981560 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:30.850303 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:30.850267 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qgstv"] Oct 31 00:02:30.850489 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:30.850415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:30.850543 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:30.850525 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:30.853371 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:30.853347 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-24nkz"] Oct 31 00:02:30.853481 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:30.853436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:30.853524 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:30.853501 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:30.856639 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:30.856612 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgbdr"] Oct 31 00:02:30.856737 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:30.856726 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:30.856837 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:30.856822 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:32.807066 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:32.807042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:32.807493 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:32.807131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:32.807493 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:32.807141 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:32.807493 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:32.807209 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:32.807493 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:32.807218 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:32.807493 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:32.807313 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:34.804638 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:34.804580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:34.805101 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:34.804776 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:34.805101 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:34.804794 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nkz" podUID="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" Oct 31 00:02:34.805101 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:34.804870 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgstv" podUID="1650ce51-15b5-4454-9c29-2a05cb986307" Oct 31 00:02:34.805101 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:34.804918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:34.805101 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:34.804982 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgbdr" podUID="af8c82f1-cb80-4dae-a124-df184eab0f73" Oct 31 00:02:36.136634 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.136612 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeReady" Oct 31 00:02:36.137087 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.136751 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:02:36.252673 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.252467 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fzds7"] Oct 31 00:02:36.254519 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.254503 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rgl6b"] Oct 31 00:02:36.254664 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.254646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fzds7" Oct 31 00:02:36.256034 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.256019 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-czswt"] Oct 31 00:02:36.256159 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.256144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" Oct 31 00:02:36.257759 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.257745 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-czswt" Oct 31 00:02:36.296567 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.296543 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7rj9b"] Oct 31 00:02:36.297917 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.297903 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29890080-867tw"] Oct 31 00:02:36.298033 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.298016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.299469 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.299452 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw"] Oct 31 00:02:36.299559 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.299545 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:36.301275 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.301259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:36.305851 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.305830 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-94bcddf7f-vbt59"] Oct 31 00:02:36.308317 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.308300 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4"] Oct 31 00:02:36.308497 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.308483 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.310567 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.310551 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl"] Oct 31 00:02:36.310698 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.310678 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.311982 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.311966 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c"] Oct 31 00:02:36.312090 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.312074 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" Oct 31 00:02:36.313442 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.313428 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-wqpvv"] Oct 31 00:02:36.313536 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.313523 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.314996 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.314982 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4"] Oct 31 00:02:36.315106 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.315091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.316467 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.316453 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-869c89c69d-stxfd"] Oct 31 00:02:36.316554 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.316542 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:36.317886 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.317872 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.347818 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.347801 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-794c4fd7c9-8n2f4"] Oct 31 00:02:36.349122 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.349101 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9wfdj"] Oct 31 00:02:36.349489 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.349474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.350793 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.350779 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd"] Oct 31 00:02:36.350902 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.350889 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:36.352722 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.352706 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2hhtz"] Oct 31 00:02:36.352835 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.352821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" Oct 31 00:02:36.354009 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.353990 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.354079 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.354007 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.354329 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.354310 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-62mlb"] Oct 31 00:02:36.354424 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.354407 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.355827 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.355812 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fzds7"] Oct 31 00:02:36.355899 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.355869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:36.356179 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.356160 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 00:02:36.356179 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.356169 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 00:02:36.356285 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.356233 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-pc4hm\"" Oct 31 00:02:36.361725 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.361702 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 00:02:36.361810 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.361775 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 00:02:36.367700 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.367683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxlv\" (UniqueName: \"kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:36.367790 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.367709 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-service-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.367790 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.367728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-snapshots\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.367790 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.367746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flhs\" (UniqueName: \"kubernetes.io/projected/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-kube-api-access-9flhs\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.367790 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.367765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqv9c\" (UniqueName: \"kubernetes.io/projected/69851eab-3b0f-4805-b6fe-2e58e42c21b0-kube-api-access-nqv9c\") pod \"kas-connection-checker-685895c7c9-czswt\" (UID: \"69851eab-3b0f-4805-b6fe-2e58e42c21b0\") " pod="kube-system/kas-connection-checker-685895c7c9-czswt" Oct 31 00:02:36.367928 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.367810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-serving-cert\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.367928 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.367892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:36.367999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.367936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn7pk\" (UniqueName: \"kubernetes.io/projected/143a4be9-3aa9-4329-88a7-71e995e72ca8-kube-api-access-pn7pk\") pod \"kas-connection-checker-685895c7c9-fzds7\" (UID: \"143a4be9-3aa9-4329-88a7-71e995e72ca8\") " pod="kube-system/kas-connection-checker-685895c7c9-fzds7" Oct 31 00:02:36.367999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.367957 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.368084 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.368045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-tmp\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.368135 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.368112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpb7\" (UniqueName: \"kubernetes.io/projected/584395de-91ba-4df6-84c3-0320c0d81cd6-kube-api-access-tnpb7\") pod \"kas-connection-checker-685895c7c9-rgl6b\" (UID: \"584395de-91ba-4df6-84c3-0320c0d81cd6\") " pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" Oct 31 00:02:36.385463 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.385441 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-czswt"] Oct 31 00:02:36.386265 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.386242 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl"] Oct 31 00:02:36.392610 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.391418 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-867tw"] Oct 31 00:02:36.392610 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.391467 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c"] Oct 31 00:02:36.392610 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.391490 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rgl6b"] Oct 31 00:02:36.393175 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.393158 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw"] Oct 31 00:02:36.394927 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.394896 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4"] Oct 31 00:02:36.394927 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.394947 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62mlb"] Oct 31 00:02:36.398905 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.398885 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-794c4fd7c9-8n2f4"] Oct 31 00:02:36.402206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.402172 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7rj9b"] Oct 31 00:02:36.408819 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.408783 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.408948 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.408844 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.408948 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.408877 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.408948 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.408877 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 00:02:36.408948 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.408914 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.408948 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.408917 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.408948 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.408949 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409041 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409082 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409089 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409111 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409125 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409132 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409111 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409163 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409262 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409289 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409353 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.409448 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409366 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.410107 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.409574 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.410163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.410106 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.410163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.410150 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.410163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.410155 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.410163 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.410152 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.410359 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.410106 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 00:02:36.410359 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.410178 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.464153 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.464129 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 00:02:36.469443 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-trusted-ca\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.469557 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-snapshots\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.469557 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7cq\" (UniqueName: \"kubernetes.io/projected/c0a8a723-d8db-407f-be86-055839588460-kube-api-access-gs7cq\") pod \"volume-data-source-validator-58c9f5f46d-nddrl\" (UID: \"c0a8a723-d8db-407f-be86-055839588460\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" Oct 31 00:02:36.469557 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a00d1d31-58a5-4835-ad53-56c65f5adbb0-serving-cert\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.469730 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbh75\" (UniqueName: \"kubernetes.io/projected/39778405-de8e-4ae1-8d8f-d2943dc7856b-kube-api-access-tbh75\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:36.469730 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flhs\" (UniqueName: \"kubernetes.io/projected/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-kube-api-access-9flhs\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.469730 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-serving-cert\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.469730 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626d2\" (UniqueName: \"kubernetes.io/projected/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-kube-api-access-626d2\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:36.469730 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.469950 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5xd2\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.469950 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-stats-auth\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.469950 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:36.469950 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.469950 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469872 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhc89\" (UniqueName: \"kubernetes.io/projected/2b251574-895b-4ac8-9688-3e094921637a-kube-api-access-nhc89\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.469950 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnmw9\" (UniqueName: \"kubernetes.io/projected/1d364537-0ba4-4fca-86c0-0575fd61a814-kube-api-access-wnmw9\") pod \"network-check-source-6f78d59f99-hq2vd\" (UID: \"1d364537-0ba4-4fca-86c0-0575fd61a814\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" Oct 31 00:02:36.470213 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.469969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-config\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.470213 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-config-volume\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.470213 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzsxr\" (UniqueName: \"kubernetes.io/projected/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-kube-api-access-lzsxr\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.470213 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7pk\" (UniqueName: \"kubernetes.io/projected/143a4be9-3aa9-4329-88a7-71e995e72ca8-kube-api-access-pn7pk\") pod \"kas-connection-checker-685895c7c9-fzds7\" (UID: \"143a4be9-3aa9-4329-88a7-71e995e72ca8\") " pod="kube-system/kas-connection-checker-685895c7c9-fzds7" Oct 31 00:02:36.470213 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbwmb\" (UniqueName: \"kubernetes.io/projected/af42676c-2cbc-47a8-a5a9-f38976f43064-kube-api-access-cbwmb\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.470213 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.470213 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470168 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-snapshots\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.470213 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.470445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-tmp-dir\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.470445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr4q2\" (UniqueName: \"kubernetes.io/projected/0dc5ea4f-6317-48ca-b075-5a292d72148c-kube-api-access-rr4q2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.470445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.470445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.470445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mj2x\" (UniqueName: \"kubernetes.io/projected/a00d1d31-58a5-4835-ad53-56c65f5adbb0-kube-api-access-7mj2x\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.470445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.470445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.470655 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.470655 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0dc5ea4f-6317-48ca-b075-5a292d72148c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.470655 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgft8\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.470655 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxlv\" (UniqueName: \"kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:36.470655 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-service-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.470872 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.470872 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.470872 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39778405-de8e-4ae1-8d8f-d2943dc7856b-cert\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:36.470872 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/32201f7b-9d56-4eb1-a3f4-b344b88beee9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:36.470872 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.470872 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqv9c\" (UniqueName: \"kubernetes.io/projected/69851eab-3b0f-4805-b6fe-2e58e42c21b0-kube-api-access-nqv9c\") pod \"kas-connection-checker-685895c7c9-czswt\" (UID: \"69851eab-3b0f-4805-b6fe-2e58e42c21b0\") " pod="kube-system/kas-connection-checker-685895c7c9-czswt" Oct 31 00:02:36.470872 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0dc5ea4f-6317-48ca-b075-5a292d72148c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-default-certificate\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af42676c-2cbc-47a8-a5a9-f38976f43064-serving-cert\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.470967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:36.471177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-tmp\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.471530 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471227 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-service-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.471530 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471287 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af42676c-2cbc-47a8-a5a9-f38976f43064-config\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.471530 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:36.471530 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471336 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:36.471530 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bmx\" (UniqueName: \"kubernetes.io/projected/d9a6cf79-72a1-4c08-ba7d-6b431911167a-kube-api-access-58bmx\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:36.471530 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471388 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-tmp\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.471530 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpb7\" (UniqueName: \"kubernetes.io/projected/584395de-91ba-4df6-84c3-0320c0d81cd6-kube-api-access-tnpb7\") pod \"kas-connection-checker-685895c7c9-rgl6b\" (UID: \"584395de-91ba-4df6-84c3-0320c0d81cd6\") " pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" Oct 31 00:02:36.471803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.471803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.471611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:36.473893 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.473876 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-serving-cert\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.498918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.498894 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 00:02:36.499033 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.498933 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:02:36.499033 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.499007 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 00:02:36.500484 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.500460 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:02:36.500623 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.500504 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 00:02:36.500623 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.500535 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:02:36.500623 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.500543 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:02:36.501170 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.501145 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.501954 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.501929 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hhtz"] Oct 31 00:02:36.503363 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.503344 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 00:02:36.503445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.503414 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:02:36.503445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.503422 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 00:02:36.503445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.503435 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 00:02:36.503445 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.503435 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 00:02:36.509283 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.509259 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hplth\"" Oct 31 00:02:36.509283 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.509276 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 00:02:36.509421 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.509263 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtt85\"" Oct 31 00:02:36.509421 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.509318 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xx4g5\"" Oct 31 00:02:36.509421 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.509350 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z67qm\"" Oct 31 00:02:36.509421 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.509354 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:02:36.511183 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.511164 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:02:36.511276 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.511194 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:02:36.511796 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.511784 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:02:36.517231 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.517119 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4"] Oct 31 00:02:36.518034 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.518014 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd"] Oct 31 00:02:36.518661 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.518646 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869c89c69d-stxfd"] Oct 31 00:02:36.519206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.519190 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-wqpvv"] Oct 31 00:02:36.519728 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.519716 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9wfdj"] Oct 31 00:02:36.520308 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.520293 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-94bcddf7f-vbt59"] Oct 31 00:02:36.532078 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.532058 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn7pk\" (UniqueName: \"kubernetes.io/projected/143a4be9-3aa9-4329-88a7-71e995e72ca8-kube-api-access-pn7pk\") pod \"kas-connection-checker-685895c7c9-fzds7\" (UID: \"143a4be9-3aa9-4329-88a7-71e995e72ca8\") " pod="kube-system/kas-connection-checker-685895c7c9-fzds7" Oct 31 00:02:36.532162 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.532149 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flhs\" (UniqueName: \"kubernetes.io/projected/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-kube-api-access-9flhs\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.532389 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.532374 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxlv\" (UniqueName: \"kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:36.533606 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.533571 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpb7\" (UniqueName: \"kubernetes.io/projected/584395de-91ba-4df6-84c3-0320c0d81cd6-kube-api-access-tnpb7\") pod \"kas-connection-checker-685895c7c9-rgl6b\" (UID: \"584395de-91ba-4df6-84c3-0320c0d81cd6\") " pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" Oct 31 00:02:36.534524 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.534507 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqv9c\" (UniqueName: \"kubernetes.io/projected/69851eab-3b0f-4805-b6fe-2e58e42c21b0-kube-api-access-nqv9c\") pod \"kas-connection-checker-685895c7c9-czswt\" (UID: \"69851eab-3b0f-4805-b6fe-2e58e42c21b0\") " pod="kube-system/kas-connection-checker-685895c7c9-czswt" Oct 31 00:02:36.566402 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.566384 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fzds7" Oct 31 00:02:36.571696 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.571681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" Oct 31 00:02:36.573014 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.572987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:36.573105 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-trusted-ca\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.573105 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7cq\" (UniqueName: \"kubernetes.io/projected/c0a8a723-d8db-407f-be86-055839588460-kube-api-access-gs7cq\") pod \"volume-data-source-validator-58c9f5f46d-nddrl\" (UID: \"c0a8a723-d8db-407f-be86-055839588460\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" Oct 31 00:02:36.573105 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a00d1d31-58a5-4835-ad53-56c65f5adbb0-serving-cert\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.573280 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbh75\" (UniqueName: \"kubernetes.io/projected/39778405-de8e-4ae1-8d8f-d2943dc7856b-kube-api-access-tbh75\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:36.573280 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.573147 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:36.573280 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.573226 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:37.073206218 +0000 UTC m=+30.697910875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:36.573442 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626d2\" (UniqueName: \"kubernetes.io/projected/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-kube-api-access-626d2\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:36.573442 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.573442 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5xd2\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.573442 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-stats-auth\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc89\" (UniqueName: \"kubernetes.io/projected/2b251574-895b-4ac8-9688-3e094921637a-kube-api-access-nhc89\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnmw9\" (UniqueName: \"kubernetes.io/projected/1d364537-0ba4-4fca-86c0-0575fd61a814-kube-api-access-wnmw9\") pod \"network-check-source-6f78d59f99-hq2vd\" (UID: \"1d364537-0ba4-4fca-86c0-0575fd61a814\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-config\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-config-volume\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzsxr\" (UniqueName: \"kubernetes.io/projected/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-kube-api-access-lzsxr\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbwmb\" (UniqueName: \"kubernetes.io/projected/af42676c-2cbc-47a8-a5a9-f38976f43064-kube-api-access-cbwmb\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-tmp-dir\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr4q2\" (UniqueName: \"kubernetes.io/projected/0dc5ea4f-6317-48ca-b075-5a292d72148c-kube-api-access-rr4q2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.573805 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mj2x\" (UniqueName: \"kubernetes.io/projected/a00d1d31-58a5-4835-ad53-56c65f5adbb0-kube-api-access-7mj2x\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0dc5ea4f-6317-48ca-b075-5a292d72148c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.573968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgft8\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574046 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-trusted-ca\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39778405-de8e-4ae1-8d8f-d2943dc7856b-cert\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/32201f7b-9d56-4eb1-a3f4-b344b88beee9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0dc5ea4f-6317-48ca-b075-5a292d72148c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-default-certificate\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.574388 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af42676c-2cbc-47a8-a5a9-f38976f43064-serving-cert\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af42676c-2cbc-47a8-a5a9-f38976f43064-config\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574541 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bmx\" (UniqueName: \"kubernetes.io/projected/d9a6cf79-72a1-4c08-ba7d-6b431911167a-kube-api-access-58bmx\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574951 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.575206 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.574968 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.576566 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.576543 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.576670 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.576938 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.577005 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls podName:c52f470a-aeaa-44a3-aa2d-7af9aa98e883 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:37.076986465 +0000 UTC m=+30.701691117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls") pod "dns-default-2hhtz" (UID: "c52f470a-aeaa-44a3-aa2d-7af9aa98e883") : secret "dns-default-metrics-tls" not found Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.577296 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af42676c-2cbc-47a8-a5a9-f38976f43064-config\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.577317 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/32201f7b-9d56-4eb1-a3f4-b344b88beee9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.577343 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.577398 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:37.077382445 +0000 UTC m=+30.702087087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.577401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-czswt" Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.577470 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.577507 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:37.077494825 +0000 UTC m=+30.702199462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : secret "router-metrics-certs-default" not found Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.577656 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:37.077641615 +0000 UTC m=+30.702346277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : configmap references non-existent config key: service-ca.crt Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.579155 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.579253 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.579268 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.579351 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:37.079334157 +0000 UTC m=+30.704038799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:36.580894 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.579890 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:02:36.582240 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.579934 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls podName:d9a6cf79-72a1-4c08-ba7d-6b431911167a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:37.079918978 +0000 UTC m=+30.704623629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4fbn4" (UID: "d9a6cf79-72a1-4c08-ba7d-6b431911167a") : secret "samples-operator-tls" not found Oct 31 00:02:36.582240 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.580986 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-default-certificate\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.582240 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.581763 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:36.582240 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.581779 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:36.582240 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:36.581822 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:37.081807683 +0000 UTC m=+30.706512338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:36.582240 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.582205 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-config-volume\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.582240 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.582228 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.582794 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.582750 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-tmp-dir\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.585628 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.585425 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af42676c-2cbc-47a8-a5a9-f38976f43064-serving-cert\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.586630 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.585750 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0dc5ea4f-6317-48ca-b075-5a292d72148c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.586630 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.585759 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.586630 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.585977 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.586630 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.586155 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.586886 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.586654 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-stats-auth\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.586886 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.586875 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39778405-de8e-4ae1-8d8f-d2943dc7856b-cert\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:36.587770 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.587687 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0dc5ea4f-6317-48ca-b075-5a292d72148c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.588286 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.588262 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a00d1d31-58a5-4835-ad53-56c65f5adbb0-serving-cert\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.588645 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.588440 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-config\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.588758 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.588731 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.589143 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.589097 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.589238 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.589198 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.611719 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.611518 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:36.619268 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.617292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:36.638541 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.638471 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbh75\" (UniqueName: \"kubernetes.io/projected/39778405-de8e-4ae1-8d8f-d2943dc7856b-kube-api-access-tbh75\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:36.652646 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.652515 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mj2x\" (UniqueName: \"kubernetes.io/projected/a00d1d31-58a5-4835-ad53-56c65f5adbb0-kube-api-access-7mj2x\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.657189 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.655762 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626d2\" (UniqueName: \"kubernetes.io/projected/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-kube-api-access-626d2\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:36.657189 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.656344 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5xd2\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.657189 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.656894 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnmw9\" (UniqueName: \"kubernetes.io/projected/1d364537-0ba4-4fca-86c0-0575fd61a814-kube-api-access-wnmw9\") pod \"network-check-source-6f78d59f99-hq2vd\" (UID: \"1d364537-0ba4-4fca-86c0-0575fd61a814\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" Oct 31 00:02:36.657189 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.657007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:36.663396 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.663363 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bmx\" (UniqueName: \"kubernetes.io/projected/d9a6cf79-72a1-4c08-ba7d-6b431911167a-kube-api-access-58bmx\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:36.663494 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.663402 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:36.666798 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.665737 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr4q2\" (UniqueName: \"kubernetes.io/projected/0dc5ea4f-6317-48ca-b075-5a292d72148c-kube-api-access-rr4q2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.666798 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.666105 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.666798 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.666391 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhc89\" (UniqueName: \"kubernetes.io/projected/2b251574-895b-4ac8-9688-3e094921637a-kube-api-access-nhc89\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:36.666798 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.666723 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbwmb\" (UniqueName: \"kubernetes.io/projected/af42676c-2cbc-47a8-a5a9-f38976f43064-kube-api-access-cbwmb\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.667232 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.666801 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgft8\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:36.668641 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.668029 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzsxr\" (UniqueName: \"kubernetes.io/projected/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-kube-api-access-lzsxr\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:36.669197 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.669104 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7cq\" (UniqueName: \"kubernetes.io/projected/c0a8a723-d8db-407f-be86-055839588460-kube-api-access-gs7cq\") pod \"volume-data-source-validator-58c9f5f46d-nddrl\" (UID: \"c0a8a723-d8db-407f-be86-055839588460\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" Oct 31 00:02:36.682102 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.681661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" Oct 31 00:02:36.703727 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.703696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:36.807624 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.804815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:36.807624 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.805295 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:36.820845 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.818989 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:36.859563 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.859536 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7lc6\"" Oct 31 00:02:36.872637 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.872618 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pp2sk\"" Oct 31 00:02:36.872732 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.872641 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:02:36.873155 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.873141 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:02:36.873195 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.873157 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x2zk\"" Oct 31 00:02:36.934078 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.934014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:36.941810 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.941793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" Oct 31 00:02:36.949260 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.949224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:36.975706 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.975519 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-czswt"] Oct 31 00:02:36.979727 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.979661 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rgl6b"] Oct 31 00:02:36.982178 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:36.982153 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fzds7"] Oct 31 00:02:37.002867 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:37.000828 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584395de_91ba_4df6_84c3_0320c0d81cd6.slice/crio-8af74d1dca5dbc47f4c725de5bcb8f6b235ea2322baaf9b3897c50c8a9230d8c WatchSource:0}: Error finding container 8af74d1dca5dbc47f4c725de5bcb8f6b235ea2322baaf9b3897c50c8a9230d8c: Status 404 returned error can't find the container with id 8af74d1dca5dbc47f4c725de5bcb8f6b235ea2322baaf9b3897c50c8a9230d8c Oct 31 00:02:37.011542 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:37.011519 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143a4be9_3aa9_4329_88a7_71e995e72ca8.slice/crio-051a34b5b8dcbf8a2a176545a793843dce214eec0a6a7d4fe3fcf7a6f2e56213 WatchSource:0}: Error finding container 051a34b5b8dcbf8a2a176545a793843dce214eec0a6a7d4fe3fcf7a6f2e56213: Status 404 returned error can't find the container with id 051a34b5b8dcbf8a2a176545a793843dce214eec0a6a7d4fe3fcf7a6f2e56213 Oct 31 00:02:37.041533 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:37.041496 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb93632_a513_4916_8377_3e2987a32f3d.slice/crio-aaf64dda097e48686e67c23d3df1e4e9849995ae9434517fcef5adfc377bf919 WatchSource:0}: Error finding container aaf64dda097e48686e67c23d3df1e4e9849995ae9434517fcef5adfc377bf919: Status 404 returned error can't find the container with id aaf64dda097e48686e67c23d3df1e4e9849995ae9434517fcef5adfc377bf919 Oct 31 00:02:37.042307 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.042239 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-867tw"] Oct 31 00:02:37.046223 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.046162 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7rj9b"] Oct 31 00:02:37.063072 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:37.063036 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61f7ab1_7e07_4f35_9ae8_9e1012e44557.slice/crio-eabce844e1b364b46cbb3a1632d06bcce6d4cf41987ed8935083f8272c2db223 WatchSource:0}: Error finding container eabce844e1b364b46cbb3a1632d06bcce6d4cf41987ed8935083f8272c2db223: Status 404 returned error can't find the container with id eabce844e1b364b46cbb3a1632d06bcce6d4cf41987ed8935083f8272c2db223 Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.081812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.081879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.081910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.081958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.082007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.082050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.082103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.082135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082269 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082326 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls podName:d9a6cf79-72a1-4c08-ba7d-6b431911167a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:38.082308195 +0000 UTC m=+31.707012836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4fbn4" (UID: "d9a6cf79-72a1-4c08-ba7d-6b431911167a") : secret "samples-operator-tls" not found Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082394 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082429 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:38.082418106 +0000 UTC m=+31.707122748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082487 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082516 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:38.082506179 +0000 UTC m=+31.707210818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:37.085270 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082571 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082632 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:38.082621413 +0000 UTC m=+31.707326050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : secret "router-metrics-certs-default" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082702 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:38.082692466 +0000 UTC m=+31.707397104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : configmap references non-existent config key: service-ca.crt Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082768 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082778 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082808 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:38.082798742 +0000 UTC m=+31.707503382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082867 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082877 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082905 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:38.082894752 +0000 UTC m=+31.707599390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082962 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:37.082990 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls podName:c52f470a-aeaa-44a3-aa2d-7af9aa98e883 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:38.082981288 +0000 UTC m=+31.707685927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls") pod "dns-default-2hhtz" (UID: "c52f470a-aeaa-44a3-aa2d-7af9aa98e883") : secret "dns-default-metrics-tls" not found Oct 31 00:02:37.086108 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.083057 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-wqpvv"] Oct 31 00:02:37.110231 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.110159 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62mlb"] Oct 31 00:02:37.120568 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.115208 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd"] Oct 31 00:02:37.125164 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:37.125052 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d364537_0ba4_4fca_86c0_0575fd61a814.slice/crio-70dadf8838413f441a84e41f6674b8ba2ab7ff3d65f496b26ad4a1343ec04814 WatchSource:0}: Error finding container 70dadf8838413f441a84e41f6674b8ba2ab7ff3d65f496b26ad4a1343ec04814: Status 404 returned error can't find the container with id 70dadf8838413f441a84e41f6674b8ba2ab7ff3d65f496b26ad4a1343ec04814 Oct 31 00:02:37.125627 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:37.125478 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39778405_de8e_4ae1_8d8f_d2943dc7856b.slice/crio-bd0bab954cc693174b54c365bd57c39dc7f4f1c0b1e120e3c81c9c2dfb5a450c WatchSource:0}: Error finding container bd0bab954cc693174b54c365bd57c39dc7f4f1c0b1e120e3c81c9c2dfb5a450c: Status 404 returned error can't find the container with id bd0bab954cc693174b54c365bd57c39dc7f4f1c0b1e120e3c81c9c2dfb5a450c Oct 31 00:02:37.306285 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.306112 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4"] Oct 31 00:02:37.308508 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:37.308478 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf42676c_2cbc_47a8_a5a9_f38976f43064.slice/crio-53c5bfff2feb2d51eab3e1af8a0e65353ab452b09fd1ff926cf4f8f31d2ae85b WatchSource:0}: Error finding container 53c5bfff2feb2d51eab3e1af8a0e65353ab452b09fd1ff926cf4f8f31d2ae85b: Status 404 returned error can't find the container with id 53c5bfff2feb2d51eab3e1af8a0e65353ab452b09fd1ff926cf4f8f31d2ae85b Oct 31 00:02:37.320420 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.320401 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl"] Oct 31 00:02:37.321746 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:37.321725 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a8a723_d8db_407f_be86_055839588460.slice/crio-8a5c74d50b848e85133d216cef218e3a69da9256d9389d6847dbd8814d7f0dbb WatchSource:0}: Error finding container 8a5c74d50b848e85133d216cef218e3a69da9256d9389d6847dbd8814d7f0dbb: Status 404 returned error can't find the container with id 8a5c74d50b848e85133d216cef218e3a69da9256d9389d6847dbd8814d7f0dbb Oct 31 00:02:37.342938 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.342919 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c"] Oct 31 00:02:37.343776 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:37.343755 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc5ea4f_6317_48ca_b075_5a292d72148c.slice/crio-cc88dfe403538dd32c3805c5ca4047a0acea7a98f9ff1efa238d691756205a75 WatchSource:0}: Error finding container cc88dfe403538dd32c3805c5ca4047a0acea7a98f9ff1efa238d691756205a75: Status 404 returned error can't find the container with id cc88dfe403538dd32c3805c5ca4047a0acea7a98f9ff1efa238d691756205a75 Oct 31 00:02:37.985137 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.985096 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" event={"ID":"af42676c-2cbc-47a8-a5a9-f38976f43064","Type":"ContainerStarted","Data":"53c5bfff2feb2d51eab3e1af8a0e65353ab452b09fd1ff926cf4f8f31d2ae85b"} Oct 31 00:02:37.986835 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.986800 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" event={"ID":"0dc5ea4f-6317-48ca-b075-5a292d72148c","Type":"ContainerStarted","Data":"cc88dfe403538dd32c3805c5ca4047a0acea7a98f9ff1efa238d691756205a75"} Oct 31 00:02:37.988547 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.988511 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-867tw" event={"ID":"8cb93632-a513-4916-8377-3e2987a32f3d","Type":"ContainerStarted","Data":"a8ec0d9ea672557c67f922f9721efa80387db4fc598f093eea60fa0efb223520"} Oct 31 00:02:37.988701 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.988555 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-867tw" event={"ID":"8cb93632-a513-4916-8377-3e2987a32f3d","Type":"ContainerStarted","Data":"aaf64dda097e48686e67c23d3df1e4e9849995ae9434517fcef5adfc377bf919"} Oct 31 00:02:37.989732 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.989710 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7rj9b" event={"ID":"f61f7ab1-7e07-4f35-9ae8-9e1012e44557","Type":"ContainerStarted","Data":"eabce844e1b364b46cbb3a1632d06bcce6d4cf41987ed8935083f8272c2db223"} Oct 31 00:02:37.991098 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.991079 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-czswt" event={"ID":"69851eab-3b0f-4805-b6fe-2e58e42c21b0","Type":"ContainerStarted","Data":"0c7d50eb34b66a51c0140aac0ae547da293c2b8fa2607d0f58eeccd5f6d36574"} Oct 31 00:02:37.991213 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.991109 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-czswt" event={"ID":"69851eab-3b0f-4805-b6fe-2e58e42c21b0","Type":"ContainerStarted","Data":"a8535088ba2febabf26f5b7f1fd42fd50572ebb28f9e1409c2e0227026f969b6"} Oct 31 00:02:37.992191 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.992170 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" event={"ID":"1d364537-0ba4-4fca-86c0-0575fd61a814","Type":"ContainerStarted","Data":"70dadf8838413f441a84e41f6674b8ba2ab7ff3d65f496b26ad4a1343ec04814"} Oct 31 00:02:37.993409 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.993373 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" event={"ID":"a00d1d31-58a5-4835-ad53-56c65f5adbb0","Type":"ContainerStarted","Data":"ec4a891f5c5714f6ef38a1c5785ae98691f80ae88bfad910e8a7f49301d7ed6b"} Oct 31 00:02:37.994608 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.994570 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" event={"ID":"c0a8a723-d8db-407f-be86-055839588460","Type":"ContainerStarted","Data":"8a5c74d50b848e85133d216cef218e3a69da9256d9389d6847dbd8814d7f0dbb"} Oct 31 00:02:37.996808 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.996752 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62mlb" event={"ID":"39778405-de8e-4ae1-8d8f-d2943dc7856b","Type":"ContainerStarted","Data":"bd0bab954cc693174b54c365bd57c39dc7f4f1c0b1e120e3c81c9c2dfb5a450c"} Oct 31 00:02:37.998294 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.998272 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" event={"ID":"584395de-91ba-4df6-84c3-0320c0d81cd6","Type":"ContainerStarted","Data":"f581e0a3246547684749af0af69358fde5a0f59ea9dd10e0029a533ab93f49fa"} Oct 31 00:02:37.998391 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.998312 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" event={"ID":"584395de-91ba-4df6-84c3-0320c0d81cd6","Type":"ContainerStarted","Data":"8af74d1dca5dbc47f4c725de5bcb8f6b235ea2322baaf9b3897c50c8a9230d8c"} Oct 31 00:02:37.999918 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.999896 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fzds7" event={"ID":"143a4be9-3aa9-4329-88a7-71e995e72ca8","Type":"ContainerStarted","Data":"a126206b92b295b0f4bb8c4ab25a2633f3d89e11a2684b9ef9901babb86c7c95"} Oct 31 00:02:38.000018 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:37.999933 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fzds7" event={"ID":"143a4be9-3aa9-4329-88a7-71e995e72ca8","Type":"ContainerStarted","Data":"051a34b5b8dcbf8a2a176545a793843dce214eec0a6a7d4fe3fcf7a6f2e56213"} Oct 31 00:02:38.090842 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.090809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:38.090985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.090932 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:38.090985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.090947 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:38.090985 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.090945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:38.091161 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091002 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.090985714 +0000 UTC m=+33.715690370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:38.091161 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.091022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:38.091161 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091056 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:02:38.091161 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.091087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:38.091161 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091104 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls podName:c52f470a-aeaa-44a3-aa2d-7af9aa98e883 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.09108925 +0000 UTC m=+33.715793901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls") pod "dns-default-2hhtz" (UID: "c52f470a-aeaa-44a3-aa2d-7af9aa98e883") : secret "dns-default-metrics-tls" not found Oct 31 00:02:38.091401 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.091169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:38.091401 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091182 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:38.091401 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.091204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:38.091401 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091219 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.091207758 +0000 UTC m=+33.715912424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:38.091401 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.091296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:38.091401 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091317 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:38.091401 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091356 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.091344418 +0000 UTC m=+33.716049069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:38.091401 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.091391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:38.091791 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091468 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:02:38.091791 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091502 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.091492805 +0000 UTC m=+33.716197476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : secret "router-metrics-certs-default" not found Oct 31 00:02:38.091791 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091569 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.091558506 +0000 UTC m=+33.716263166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : configmap references non-existent config key: service-ca.crt Oct 31 00:02:38.091791 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091629 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:38.091791 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091639 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:02:38.091791 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091644 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:38.091791 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091670 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls podName:d9a6cf79-72a1-4c08-ba7d-6b431911167a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.091660671 +0000 UTC m=+33.716365347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4fbn4" (UID: "d9a6cf79-72a1-4c08-ba7d-6b431911167a") : secret "samples-operator-tls" not found Oct 31 00:02:38.091791 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:38.091684 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.091676337 +0000 UTC m=+33.716380977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:38.801967 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.800985 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-czswt" podStartSLOduration=347.800966938 podStartE2EDuration="5m47.800966938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:38.682356294 +0000 UTC m=+32.307060955" watchObservedRunningTime="2026-10-31 00:02:38.800966938 +0000 UTC m=+32.425671599" Oct 31 00:02:38.921194 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.920506 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fzds7" podStartSLOduration=348.920486617 podStartE2EDuration="5m48.920486617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:38.918571301 +0000 UTC m=+32.543275962" watchObservedRunningTime="2026-10-31 00:02:38.920486617 +0000 UTC m=+32.545191278" Oct 31 00:02:38.921194 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:38.920625 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29890080-867tw" podStartSLOduration=158.920620255 podStartE2EDuration="2m38.920620255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:38.801835005 +0000 UTC m=+32.426539657" watchObservedRunningTime="2026-10-31 00:02:38.920620255 +0000 UTC m=+32.545324914" Oct 31 00:02:39.096854 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:39.096518 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" podStartSLOduration=348.096499178 podStartE2EDuration="5m48.096499178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:39.094671374 +0000 UTC m=+32.719376046" watchObservedRunningTime="2026-10-31 00:02:39.096499178 +0000 UTC m=+32.721203841" Oct 31 00:02:39.814487 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:39.814047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:39.814487 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:39.814166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:39.814487 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:39.814399 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:02:39.814487 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:39.814461 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:11.81444286 +0000 UTC m=+65.439147502 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : secret "metrics-daemon-secret" not found Oct 31 00:02:39.816586 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:39.816564 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:39.842461 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:39.842417 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:39.917054 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:39.915557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:39.942754 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:39.942379 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:40.013011 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.012333 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" event={"ID":"c0a8a723-d8db-407f-be86-055839588460","Type":"ContainerStarted","Data":"c366d214b3bcf1c4a19bf44c985554f8a9531f0f176424fb5d071464ef9a0b70"} Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.117758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.117822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.117856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.117886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.117935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.117996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.118028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.118072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118241 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.118222426 +0000 UTC m=+37.742927066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : configmap references non-existent config key: service-ca.crt Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118679 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118695 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118741 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.118727074 +0000 UTC m=+37.743431716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118805 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118814 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118841 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.118830953 +0000 UTC m=+37.743535594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:40.120030 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118898 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:02:40.120985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118929 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls podName:c52f470a-aeaa-44a3-aa2d-7af9aa98e883 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.118919244 +0000 UTC m=+37.743623888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls") pod "dns-default-2hhtz" (UID: "c52f470a-aeaa-44a3-aa2d-7af9aa98e883") : secret "dns-default-metrics-tls" not found Oct 31 00:02:40.120985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.118986 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:02:40.120985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.119016 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls podName:d9a6cf79-72a1-4c08-ba7d-6b431911167a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.119005887 +0000 UTC m=+37.743710531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4fbn4" (UID: "d9a6cf79-72a1-4c08-ba7d-6b431911167a") : secret "samples-operator-tls" not found Oct 31 00:02:40.120985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.119068 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:40.120985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.119098 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.119088508 +0000 UTC m=+37.743793152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:40.120985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.119839 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:40.120985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.119883 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.11986963 +0000 UTC m=+37.744574271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:40.120985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.119956 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:02:40.120985 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:40.119988 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.119977546 +0000 UTC m=+37.744682187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : secret "router-metrics-certs-default" not found Oct 31 00:02:40.131225 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.131170 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" podStartSLOduration=316.760126466 podStartE2EDuration="5m19.131155477s" totalImagesPullingTime="2.371029011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:40.129756986 +0000 UTC m=+33.754461647" watchObservedRunningTime="2026-10-31 00:02:40.131155477 +0000 UTC m=+33.755860141" Oct 31 00:02:40.147061 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.147037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:40.242376 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.242342 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-24nkz"] Oct 31 00:02:40.258705 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:40.256835 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ae8cb8_c2ec_4476_bc1d_c13e3ae10454.slice/crio-f332a2a05f016bc90997d92fc76adf1b84e0f7f688ec750805aeef931ea06528 WatchSource:0}: Error finding container f332a2a05f016bc90997d92fc76adf1b84e0f7f688ec750805aeef931ea06528: Status 404 returned error can't find the container with id f332a2a05f016bc90997d92fc76adf1b84e0f7f688ec750805aeef931ea06528 Oct 31 00:02:40.564577 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:40.564455 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qgstv"] Oct 31 00:02:40.569084 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:40.569054 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1650ce51_15b5_4454_9c29_2a05cb986307.slice/crio-e4deb00504d59bdbe4b5f253d92328842bd7c772a7d173ea8cbbb027ad1eb695 WatchSource:0}: Error finding container e4deb00504d59bdbe4b5f253d92328842bd7c772a7d173ea8cbbb027ad1eb695: Status 404 returned error can't find the container with id e4deb00504d59bdbe4b5f253d92328842bd7c772a7d173ea8cbbb027ad1eb695 Oct 31 00:02:41.019611 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:41.019464 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-24nkz" event={"ID":"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454","Type":"ContainerStarted","Data":"f332a2a05f016bc90997d92fc76adf1b84e0f7f688ec750805aeef931ea06528"} Oct 31 00:02:41.022121 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:41.022095 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qgstv" event={"ID":"1650ce51-15b5-4454-9c29-2a05cb986307","Type":"ContainerStarted","Data":"e4deb00504d59bdbe4b5f253d92328842bd7c772a7d173ea8cbbb027ad1eb695"} Oct 31 00:02:42.029752 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:42.029715 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7rj9b" event={"ID":"f61f7ab1-7e07-4f35-9ae8-9e1012e44557","Type":"ContainerStarted","Data":"7819c510f427dddc89e1804abfbb46654f78be5a9f672fc143a946953efdcda3"} Oct 31 00:02:42.036009 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:42.034011 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" event={"ID":"a00d1d31-58a5-4835-ad53-56c65f5adbb0","Type":"ContainerStarted","Data":"faa1130855b8f46f6cfeea08d0acce795e30a14013fd83301b9ab643a48e93ca"} Oct 31 00:02:42.036156 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:42.036029 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:42.036501 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:42.036424 2580 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-wqpvv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.21:8443/readyz\": dial tcp 10.133.0.21:8443: connect: connection refused" start-of-body= Oct 31 00:02:42.036638 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:42.036557 2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.21:8443/readyz\": dial tcp 10.133.0.21:8443: connect: connection refused" Oct 31 00:02:42.150622 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:42.150521 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-7rj9b" podStartSLOduration=316.333529922 podStartE2EDuration="5m21.1505056s" totalImagesPullingTime="4.816975678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:42.149692434 +0000 UTC m=+35.774397095" watchObservedRunningTime="2026-10-31 00:02:42.1505056 +0000 UTC m=+35.775210259" Oct 31 00:02:42.269138 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:42.269056 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podStartSLOduration=316.511233812 podStartE2EDuration="5m21.269040224s" totalImagesPullingTime="4.757806412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:42.268622503 +0000 UTC m=+35.893327163" watchObservedRunningTime="2026-10-31 00:02:42.269040224 +0000 UTC m=+35.893744913" Oct 31 00:02:43.042453 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.042364 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" event={"ID":"0dc5ea4f-6317-48ca-b075-5a292d72148c","Type":"ContainerStarted","Data":"cc855b40a56c90efa6fc19ea584ba7279ae54fef463b09c6f3cdf7fd49a31a7a"} Oct 31 00:02:43.046084 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.045645 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/0.log" Oct 31 00:02:43.046084 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.045685 2580 generic.go:356] "Generic (PLEG): container finished" podID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" containerID="faa1130855b8f46f6cfeea08d0acce795e30a14013fd83301b9ab643a48e93ca" exitCode=255 Oct 31 00:02:43.046084 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.045790 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" event={"ID":"a00d1d31-58a5-4835-ad53-56c65f5adbb0","Type":"ContainerDied","Data":"faa1130855b8f46f6cfeea08d0acce795e30a14013fd83301b9ab643a48e93ca"} Oct 31 00:02:43.046873 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.046444 2580 scope.go:122] "RemoveContainer" containerID="faa1130855b8f46f6cfeea08d0acce795e30a14013fd83301b9ab643a48e93ca" Oct 31 00:02:43.049925 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.049863 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" event={"ID":"af42676c-2cbc-47a8-a5a9-f38976f43064","Type":"ContainerStarted","Data":"bd9f71749ee877d1716dd933921096fdba641ce51bf8d624a16c921522879d7b"} Oct 31 00:02:43.280573 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.280509 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" podStartSLOduration=317.223792917 podStartE2EDuration="5m22.280489742s" totalImagesPullingTime="5.056696825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:43.279463946 +0000 UTC m=+36.904168607" watchObservedRunningTime="2026-10-31 00:02:43.280489742 +0000 UTC m=+36.905194403" Oct 31 00:02:43.281730 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.281688 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" podStartSLOduration=317.236556244 podStartE2EDuration="5m22.281672435s" totalImagesPullingTime="5.045116191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:43.161907686 +0000 UTC m=+36.786612367" watchObservedRunningTime="2026-10-31 00:02:43.281672435 +0000 UTC m=+36.906377115" Oct 31 00:02:43.737204 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.737167 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hzjx9"] Oct 31 00:02:43.756644 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.755846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.761021 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.760994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb324747-4db5-4661-a18e-35933b8ad5ad-crio-socket\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.761144 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.761068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.761144 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.761126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb324747-4db5-4661-a18e-35933b8ad5ad-data-volume\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.761251 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.761180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2pt\" (UniqueName: \"kubernetes.io/projected/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-api-access-9x2pt\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.761251 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.761211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.830523 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.830486 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzjx9"] Oct 31 00:02:43.861988 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.861951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb324747-4db5-4661-a18e-35933b8ad5ad-crio-socket\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.862134 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.862040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.862134 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.862086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb324747-4db5-4661-a18e-35933b8ad5ad-data-volume\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.862134 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.862090 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb324747-4db5-4661-a18e-35933b8ad5ad-crio-socket\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.862297 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.862154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2pt\" (UniqueName: \"kubernetes.io/projected/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-api-access-9x2pt\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.862297 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.862188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.862392 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.862378 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb324747-4db5-4661-a18e-35933b8ad5ad-data-volume\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:43.881890 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.881864 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:02:43.882132 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.881923 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:02:43.882225 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.882128 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4gwp\"" Oct 31 00:02:43.883150 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:43.883130 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:02:43.883289 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:43.883204 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.383188258 +0000 UTC m=+38.007892909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : secret "insights-runtime-extractor-tls" not found Oct 31 00:02:43.883289 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:43.883141 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:02:43.883289 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:43.883252 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.383240102 +0000 UTC m=+38.007944768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : configmap "kube-rbac-proxy" not found Oct 31 00:02:43.929133 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:43.929082 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2pt\" (UniqueName: \"kubernetes.io/projected/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-api-access-9x2pt\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:44.055896 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.055876 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/1.log" Oct 31 00:02:44.056396 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.056382 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/0.log" Oct 31 00:02:44.056513 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.056419 2580 generic.go:356] "Generic (PLEG): container finished" podID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" containerID="fc82fc114325880ddef36554f3c84dca7816e34a56beb8db5c5fc2e991a794dd" exitCode=255 Oct 31 00:02:44.056513 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.056497 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" event={"ID":"a00d1d31-58a5-4835-ad53-56c65f5adbb0","Type":"ContainerDied","Data":"fc82fc114325880ddef36554f3c84dca7816e34a56beb8db5c5fc2e991a794dd"} Oct 31 00:02:44.056639 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.056568 2580 scope.go:122] "RemoveContainer" containerID="faa1130855b8f46f6cfeea08d0acce795e30a14013fd83301b9ab643a48e93ca" Oct 31 00:02:44.056818 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.056802 2580 scope.go:122] "RemoveContainer" containerID="fc82fc114325880ddef36554f3c84dca7816e34a56beb8db5c5fc2e991a794dd" Oct 31 00:02:44.057038 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.057016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:02:44.060867 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.060832 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62mlb" event={"ID":"39778405-de8e-4ae1-8d8f-d2943dc7856b","Type":"ContainerStarted","Data":"d5ed1c2107e032fcbc0c022848fd5e74def3db4cd7fb3b1fb30778f33c8c4151"} Oct 31 00:02:44.164649 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.164584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:44.164812 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.164755 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:02:44.164881 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.164834 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls podName:c52f470a-aeaa-44a3-aa2d-7af9aa98e883 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.164814318 +0000 UTC m=+45.789518971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls") pod "dns-default-2hhtz" (UID: "c52f470a-aeaa-44a3-aa2d-7af9aa98e883") : secret "dns-default-metrics-tls" not found Oct 31 00:02:44.164881 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.164868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:44.164999 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.164925 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:44.165049 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.164996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:44.165049 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165014 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:02:44.165049 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.165024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:44.165243 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165058 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls podName:d9a6cf79-72a1-4c08-ba7d-6b431911167a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.165044692 +0000 UTC m=+45.789749351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4fbn4" (UID: "d9a6cf79-72a1-4c08-ba7d-6b431911167a") : secret "samples-operator-tls" not found Oct 31 00:02:44.165243 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165105 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:02:44.165243 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.165112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:44.165243 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165137 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.165126281 +0000 UTC m=+45.789830952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : secret "router-metrics-certs-default" not found Oct 31 00:02:44.165243 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165177 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.165170965 +0000 UTC m=+45.789875637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : configmap references non-existent config key: service-ca.crt Oct 31 00:02:44.165243 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.165203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:44.165540 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.165265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:44.165540 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165344 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:44.165540 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165351 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:44.165540 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165371 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.165364698 +0000 UTC m=+45.790069336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:44.165540 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165395 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:44.165540 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165415 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.165410308 +0000 UTC m=+45.790114947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:44.165796 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165758 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:44.165874 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165862 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:44.165923 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165877 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:44.165923 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165912 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.165899115 +0000 UTC m=+45.790603781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:44.166021 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.165945 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.165935644 +0000 UTC m=+45.790640287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:44.298784 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.298471 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-62mlb" podStartSLOduration=2.226343429 podStartE2EDuration="8.29845422s" totalImagesPullingTime="6.072110791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:44.297062585 +0000 UTC m=+37.921767245" watchObservedRunningTime="2026-10-31 00:02:44.29845422 +0000 UTC m=+37.923158880" Oct 31 00:02:44.469803 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.469768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:44.469987 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.469880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:44.469987 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.469898 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:02:44.469987 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.469970 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:45.469948399 +0000 UTC m=+39.094653050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : configmap "kube-rbac-proxy" not found Oct 31 00:02:44.470141 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.470012 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:02:44.470141 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:44.470076 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:45.470060778 +0000 UTC m=+39.094765417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : secret "insights-runtime-extractor-tls" not found Oct 31 00:02:44.712787 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.712754 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj"] Oct 31 00:02:44.749177 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.749144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj" Oct 31 00:02:44.773029 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.773004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ssf7\" (UniqueName: \"kubernetes.io/projected/a0f9a47b-58ae-4467-94a0-fc93927af017-kube-api-access-5ssf7\") pod \"migrator-66df5d496b-hsrfj\" (UID: \"a0f9a47b-58ae-4467-94a0-fc93927af017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj" Oct 31 00:02:44.773430 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.773403 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj"] Oct 31 00:02:44.855315 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.855284 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zm4vk\"" Oct 31 00:02:44.855493 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.855454 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:44.855573 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.855537 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:02:44.873617 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.873516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ssf7\" (UniqueName: \"kubernetes.io/projected/a0f9a47b-58ae-4467-94a0-fc93927af017-kube-api-access-5ssf7\") pod \"migrator-66df5d496b-hsrfj\" (UID: \"a0f9a47b-58ae-4467-94a0-fc93927af017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj" Oct 31 00:02:44.936679 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:44.936654 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ssf7\" (UniqueName: \"kubernetes.io/projected/a0f9a47b-58ae-4467-94a0-fc93927af017-kube-api-access-5ssf7\") pod \"migrator-66df5d496b-hsrfj\" (UID: \"a0f9a47b-58ae-4467-94a0-fc93927af017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj" Oct 31 00:02:45.067238 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.067094 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" event={"ID":"1d364537-0ba4-4fca-86c0-0575fd61a814","Type":"ContainerStarted","Data":"72a1dc4d94162d5089a8d85f28015a3a3cb72143cc1f9b54e8daffe24de9283a"} Oct 31 00:02:45.069643 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.069610 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/1.log" Oct 31 00:02:45.069963 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.069948 2580 scope.go:122] "RemoveContainer" containerID="fc82fc114325880ddef36554f3c84dca7816e34a56beb8db5c5fc2e991a794dd" Oct 31 00:02:45.070151 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:45.070126 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:02:45.071893 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.071870 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qgstv" event={"ID":"1650ce51-15b5-4454-9c29-2a05cb986307","Type":"ContainerStarted","Data":"6f2216add2c382c2ad901c006c00eb6647218c0f2ec2cb4ac6f3e2f3f99b5f06"} Oct 31 00:02:45.072037 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.072007 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:45.076519 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.076497 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj" Oct 31 00:02:45.191357 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.190953 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" podStartSLOduration=314.333289859 podStartE2EDuration="5m21.190928745s" totalImagesPullingTime="6.857638886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:45.190326563 +0000 UTC m=+38.815031242" watchObservedRunningTime="2026-10-31 00:02:45.190928745 +0000 UTC m=+38.815633410" Oct 31 00:02:45.474564 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.474504 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qgstv" podStartSLOduration=34.173489568 podStartE2EDuration="38.474487875s" totalImagesPullingTime="4.300998307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:45.431502084 +0000 UTC m=+39.056206744" watchObservedRunningTime="2026-10-31 00:02:45.474487875 +0000 UTC m=+39.099192537" Oct 31 00:02:45.475074 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.475042 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj"] Oct 31 00:02:45.478871 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.478844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:45.478994 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:45.478971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:45.479060 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:45.478975 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:02:45.479112 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:45.479060 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:02:45.479112 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:45.479106 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:47.479088049 +0000 UTC m=+41.103792706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : configmap "kube-rbac-proxy" not found Oct 31 00:02:45.479208 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:45.479128 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:47.479118274 +0000 UTC m=+41.103822912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : secret "insights-runtime-extractor-tls" not found Oct 31 00:02:45.480785 ip-10-0-141-63 kubenswrapper[2580]: W1031 00:02:45.480762 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f9a47b_58ae_4467_94a0_fc93927af017.slice/crio-00e1004e761e6b65ba3351a61bd00e252e687f111ced112c25ebb34520cac474 WatchSource:0}: Error finding container 00e1004e761e6b65ba3351a61bd00e252e687f111ced112c25ebb34520cac474: Status 404 returned error can't find the container with id 00e1004e761e6b65ba3351a61bd00e252e687f111ced112c25ebb34520cac474 Oct 31 00:02:46.077112 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:46.077038 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj" event={"ID":"a0f9a47b-58ae-4467-94a0-fc93927af017","Type":"ContainerStarted","Data":"00e1004e761e6b65ba3351a61bd00e252e687f111ced112c25ebb34520cac474"} Oct 31 00:02:46.659461 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:46.659427 2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:46.660059 ip-10-0-141-63 kubenswrapper[2580]: I1031 00:02:46.660010 2580 scope.go:122] "RemoveContainer" containerID="fc82fc114325880ddef36554f3c84dca7816e34a56beb8db5c5fc2e991a794dd" Oct 31 00:02:46.660273 ip-10-0-141-63 kubenswrapper[2580]: E1031 00:02:46.660249 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:02:47.419878 ip-10-0-141-63 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:02:47.434990 ip-10-0-141-63 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:02:47.435160 ip-10-0-141-63 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:02:47.435209 ip-10-0-141-63 systemd[1]: kubelet.service: Consumed 3.120s CPU time. Oct 31 00:02:47.446230 ip-10-0-141-63 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:02:47.554489 ip-10-0-141-63 kubenswrapper[4969]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:02:47.554489 ip-10-0-141-63 kubenswrapper[4969]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:02:47.554489 ip-10-0-141-63 kubenswrapper[4969]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:02:47.554489 ip-10-0-141-63 kubenswrapper[4969]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:02:47.556535 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556460 4969 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:47.556535 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556537 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556541 4969 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556545 4969 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556548 4969 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556552 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556555 4969 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556558 4969 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556561 4969 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556564 4969 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556566 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556569 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556571 4969 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556574 4969 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556585 4969 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556606 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556611 4969 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556615 4969 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556619 4969 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556622 4969 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556625 4969 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:47.556622 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556628 4969 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556631 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556633 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556636 4969 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556639 4969 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556641 4969 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556644 4969 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556648 4969 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556652 4969 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556655 4969 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556658 4969 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556660 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556663 4969 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556666 4969 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556668 4969 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556671 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556673 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556676 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556678 4969 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:47.557121 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556680 4969 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556683 4969 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556685 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556688 4969 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556691 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556694 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556696 4969 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556699 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556702 4969 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556705 4969 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556707 4969 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556710 4969 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556712 4969 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556718 4969 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556721 4969 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556724 4969 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556726 4969 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556729 4969 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556731 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556734 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:47.557572 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556737 4969 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556740 4969 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556742 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556745 4969 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556747 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556750 4969 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556753 4969 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556756 4969 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556758 4969 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556761 4969 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556763 4969 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556766 4969 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556768 4969 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556772 4969 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556774 4969 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556778 4969 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556782 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556785 4969 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556787 4969 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:47.558073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556790 4969 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556802 4969 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556805 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556808 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556810 4969 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556813 4969 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556817 4969 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556820 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556823 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556825 4969 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556828 4969 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556831 4969 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556834 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556836 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556838 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556841 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556843 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556846 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556848 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556851 4969 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:47.558563 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556853 4969 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556856 4969 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556858 4969 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556860 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556863 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556865 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.556868 4969 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557656 4969 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557664 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557666 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557669 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557672 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557675 4969 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557677 4969 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557680 4969 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557683 4969 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557686 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557689 4969 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557692 4969 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557695 4969 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:47.559112 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557697 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557701 4969 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557703 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557706 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557708 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557711 4969 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557713 4969 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557715 4969 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557718 4969 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557720 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557723 4969 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557725 4969 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557728 4969 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557730 4969 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557733 4969 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557736 4969 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557738 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557740 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557743 4969 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:47.559628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557745 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557747 4969 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557750 4969 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557752 4969 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557755 4969 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557757 4969 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557760 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557762 4969 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557766 4969 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557768 4969 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557771 4969 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557774 4969 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557777 4969 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557779 4969 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557782 4969 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557784 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557787 4969 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557789 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557791 4969 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557794 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:47.560091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557796 4969 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557799 4969 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557803 4969 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557807 4969 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557811 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557814 4969 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557819 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557822 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557825 4969 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557827 4969 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557830 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557832 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557835 4969 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557837 4969 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557841 4969 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557844 4969 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557847 4969 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557851 4969 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557853 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:47.560573 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557856 4969 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557859 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557864 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557867 4969 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557870 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557873 4969 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557875 4969 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557878 4969 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557880 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557883 4969 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557886 4969 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557888 4969 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557891 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557893 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557896 4969 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557898 4969 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557901 4969 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557903 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557906 4969 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:47.561153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557908 4969 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557911 4969 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557913 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557918 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557920 4969 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557923 4969 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557925 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557928 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557930 4969 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557933 4969 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557936 4969 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557939 4969 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557941 4969 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557944 4969 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557946 4969 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.557949 4969 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558518 4969 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558523 4969 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558526 4969 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:47.561707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558528 4969 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558531 4969 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558534 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558536 4969 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558539 4969 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558542 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558544 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558548 4969 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558553 4969 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558555 4969 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558558 4969 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558561 4969 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558563 4969 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558566 4969 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558569 4969 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558572 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558576 4969 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558579 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558582 4969 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:47.562161 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558585 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558605 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558609 4969 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558613 4969 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558617 4969 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558621 4969 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558625 4969 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558629 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558633 4969 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558636 4969 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558645 4969 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558648 4969 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558650 4969 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558653 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558655 4969 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558658 4969 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558660 4969 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558663 4969 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558665 4969 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:47.562673 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558668 4969 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558670 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558673 4969 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558675 4969 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558678 4969 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558680 4969 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558683 4969 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558685 4969 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558688 4969 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558691 4969 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558695 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558698 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558700 4969 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558704 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558706 4969 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558709 4969 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558712 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558714 4969 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558717 4969 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558719 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:47.563146 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558722 4969 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558724 4969 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558727 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558729 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558739 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558742 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558745 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558747 4969 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558750 4969 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558752 4969 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558755 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558757 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558759 4969 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558762 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558764 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558767 4969 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558769 4969 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558772 4969 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558774 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558777 4969 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:47.564039 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558779 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558782 4969 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558787 4969 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558790 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558792 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558795 4969 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558797 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558800 4969 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558802 4969 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558805 4969 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558807 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558810 4969 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558812 4969 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558815 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558817 4969 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558819 4969 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558822 4969 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558830 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558832 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:47.564699 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558835 4969 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558837 4969 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558840 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558842 4969 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558845 4969 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.558849 4969 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.558952 4969 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.558959 4969 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.558974 4969 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.558978 4969 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.558982 4969 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.558986 4969 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.558990 4969 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.558994 4969 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.558997 4969 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559002 4969 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559006 4969 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559009 4969 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559012 4969 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559015 4969 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559017 4969 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559020 4969 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559023 4969 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:02:47.565165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559026 4969 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559032 4969 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559035 4969 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559038 4969 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559041 4969 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559044 4969 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559048 4969 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559051 4969 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559060 4969 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559063 4969 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559066 4969 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559069 4969 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559072 4969 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559075 4969 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559078 4969 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559082 4969 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559085 4969 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559088 4969 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559090 4969 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559093 4969 flags.go:64] FLAG: --enable-server="true" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559096 4969 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559103 4969 flags.go:64] FLAG: --event-burst="100" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559106 4969 flags.go:64] FLAG: --event-qps="50" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559109 4969 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:02:47.565742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559116 4969 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559119 4969 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559123 4969 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559126 4969 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559129 4969 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559132 4969 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559135 4969 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559138 4969 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559141 4969 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559144 4969 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559147 4969 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559149 4969 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559152 4969 flags.go:64] FLAG: --feature-gates="" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559156 4969 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559159 4969 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559162 4969 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559165 4969 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559176 4969 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559179 4969 flags.go:64] FLAG: --help="false" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559182 4969 flags.go:64] FLAG: --hostname-override="ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559185 4969 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559188 4969 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559191 4969 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:02:47.566346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559194 4969 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559198 4969 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559200 4969 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559203 4969 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559206 4969 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559209 4969 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559212 4969 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559215 4969 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559218 4969 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559223 4969 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559225 4969 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559229 4969 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559231 4969 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559234 4969 flags.go:64] FLAG: --lock-file="" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559237 4969 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559240 4969 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559243 4969 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559248 4969 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559251 4969 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559254 4969 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559257 4969 flags.go:64] FLAG: --logging-format="text" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559259 4969 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559263 4969 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559265 4969 flags.go:64] FLAG: --manifest-url="" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559268 4969 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:02:47.566976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559273 4969 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559276 4969 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559286 4969 flags.go:64] FLAG: --max-pods="110" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559289 4969 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559292 4969 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559295 4969 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559298 4969 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559301 4969 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559303 4969 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559306 4969 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559313 4969 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559316 4969 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559318 4969 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559321 4969 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559324 4969 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559327 4969 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559332 4969 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559335 4969 flags.go:64] FLAG: --port="10250" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559338 4969 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559341 4969 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-073c3d8149f0f39eb" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559344 4969 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559347 4969 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559350 4969 flags.go:64] FLAG: --register-node="true" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559353 4969 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559357 4969 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559360 4969 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:02:47.567582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559362 4969 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559365 4969 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559369 4969 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559372 4969 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559375 4969 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559378 4969 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559380 4969 flags.go:64] FLAG: --runonce="false" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559383 4969 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559386 4969 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559396 4969 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559405 4969 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559408 4969 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559411 4969 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559414 4969 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559417 4969 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559420 4969 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559423 4969 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559426 4969 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559429 4969 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559432 4969 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559435 4969 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559437 4969 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559445 4969 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559448 4969 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559451 4969 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:02:47.568258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559457 4969 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559460 4969 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559462 4969 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559465 4969 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559468 4969 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559471 4969 flags.go:64] FLAG: --v="2" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559475 4969 flags.go:64] FLAG: --version="false" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559479 4969 flags.go:64] FLAG: --vmodule="" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559483 4969 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.559486 4969 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559682 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559686 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559691 4969 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559695 4969 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559698 4969 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559701 4969 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559704 4969 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559712 4969 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559715 4969 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559718 4969 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559721 4969 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559724 4969 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:47.568897 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559726 4969 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559729 4969 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559731 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559734 4969 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559737 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559739 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559742 4969 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559747 4969 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559750 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559753 4969 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559756 4969 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559758 4969 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559761 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559764 4969 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559766 4969 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559769 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559771 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559774 4969 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559777 4969 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:47.569428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559779 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559782 4969 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559784 4969 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559787 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559789 4969 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559792 4969 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559795 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559797 4969 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559800 4969 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559809 4969 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559812 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559815 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559817 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559820 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559822 4969 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559825 4969 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559827 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559830 4969 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559833 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559835 4969 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:47.570017 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559840 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559843 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559845 4969 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559848 4969 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559850 4969 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559853 4969 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559855 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559858 4969 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559860 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559863 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559865 4969 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559868 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559870 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559873 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559875 4969 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559878 4969 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559880 4969 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559883 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559885 4969 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559887 4969 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:47.570582 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559890 4969 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559892 4969 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559901 4969 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559904 4969 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559906 4969 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559909 4969 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559911 4969 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559914 4969 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559916 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559918 4969 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559921 4969 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559924 4969 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559929 4969 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559931 4969 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559934 4969 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559936 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559939 4969 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559941 4969 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559944 4969 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:47.571450 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559947 4969 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559949 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559952 4969 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559954 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559957 4969 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559959 4969 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559962 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559964 4969 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559968 4969 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559972 4969 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559975 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559978 4969 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559981 4969 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559983 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559986 4969 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.559989 4969 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:47.572093 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.560021 4969 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560061 4969 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560066 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560069 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560071 4969 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560074 4969 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560076 4969 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560079 4969 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560081 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560086 4969 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560090 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560092 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560095 4969 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560098 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560101 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560103 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560106 4969 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560108 4969 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560111 4969 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560113 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:47.572479 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560116 4969 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560118 4969 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560121 4969 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560123 4969 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560126 4969 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560129 4969 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560131 4969 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560134 4969 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560136 4969 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560139 4969 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560141 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560144 4969 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560146 4969 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560154 4969 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560157 4969 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560159 4969 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560161 4969 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560164 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560166 4969 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:47.573109 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560169 4969 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560171 4969 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560174 4969 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560178 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560181 4969 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560183 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560186 4969 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560188 4969 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560191 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560193 4969 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560197 4969 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560200 4969 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560202 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560205 4969 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560207 4969 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560210 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560212 4969 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560215 4969 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560218 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:47.573816 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560220 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560224 4969 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560228 4969 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560231 4969 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560233 4969 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560236 4969 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560238 4969 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560241 4969 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560249 4969 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560252 4969 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560255 4969 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560257 4969 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560260 4969 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560262 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560265 4969 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560268 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560273 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560275 4969 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560278 4969 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:47.574405 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560280 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560283 4969 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560286 4969 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560288 4969 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560291 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560293 4969 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560296 4969 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560299 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560301 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560304 4969 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560307 4969 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560309 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560312 4969 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560314 4969 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560317 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560319 4969 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560322 4969 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560324 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560327 4969 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560330 4969 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:47.574894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560332 4969 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560335 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560343 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560345 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560348 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560350 4969 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560353 4969 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560355 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560358 4969 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.560362 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.565750 4969 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.565765 4969 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.565782 4969 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.565787 4969 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565849 4969 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565854 4969 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565857 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565860 4969 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565862 4969 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565866 4969 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:47.575395 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565869 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565871 4969 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565874 4969 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565877 4969 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565880 4969 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565883 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565885 4969 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565888 4969 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565890 4969 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565893 4969 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565896 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565899 4969 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565901 4969 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565904 4969 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565906 4969 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565909 4969 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565911 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565914 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565916 4969 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565919 4969 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:47.575916 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565922 4969 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565926 4969 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565928 4969 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565931 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565933 4969 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565936 4969 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565939 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565943 4969 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565947 4969 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565951 4969 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565954 4969 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565958 4969 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565962 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565965 4969 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565967 4969 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565970 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565972 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565975 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565977 4969 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:47.576393 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565980 4969 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565984 4969 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565987 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565990 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565992 4969 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565995 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.565997 4969 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566000 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566002 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566005 4969 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566008 4969 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566010 4969 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566013 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566015 4969 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566018 4969 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566020 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566023 4969 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566025 4969 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566028 4969 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:47.576868 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566030 4969 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566032 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566035 4969 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566037 4969 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566040 4969 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566043 4969 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566045 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566047 4969 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566050 4969 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566052 4969 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566055 4969 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566058 4969 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566060 4969 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566062 4969 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566065 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566068 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566070 4969 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566073 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566075 4969 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566078 4969 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:47.577379 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566081 4969 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566083 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566086 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566088 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566091 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566093 4969 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566096 4969 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566098 4969 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566100 4969 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566103 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566106 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566108 4969 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566111 4969 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566113 4969 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566115 4969 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566118 4969 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566121 4969 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566123 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566126 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:47.578223 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566128 4969 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566130 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566133 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.566160 4969 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566305 4969 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566310 4969 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566313 4969 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566316 4969 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566318 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566324 4969 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566326 4969 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566329 4969 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566332 4969 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566334 4969 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566336 4969 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566339 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:47.578840 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566341 4969 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566344 4969 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566346 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566349 4969 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566352 4969 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566354 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566357 4969 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566359 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566362 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566365 4969 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566367 4969 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566370 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566372 4969 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566374 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566378 4969 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566382 4969 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566385 4969 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566388 4969 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566390 4969 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566393 4969 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:47.579222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566395 4969 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566398 4969 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566400 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566403 4969 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566405 4969 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566407 4969 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566411 4969 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566413 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566416 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566418 4969 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566421 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566423 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566426 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566428 4969 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566433 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566435 4969 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566438 4969 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566440 4969 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566443 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:47.579774 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566445 4969 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566447 4969 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566450 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566453 4969 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566455 4969 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566458 4969 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566460 4969 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566463 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566465 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566468 4969 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566470 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566472 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566475 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566477 4969 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566480 4969 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566482 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566485 4969 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566487 4969 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566489 4969 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:47.580302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566492 4969 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566495 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566498 4969 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566500 4969 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566503 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566506 4969 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566508 4969 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566510 4969 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566514 4969 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566517 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566519 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566522 4969 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566524 4969 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566527 4969 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566529 4969 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566531 4969 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566534 4969 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566537 4969 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566539 4969 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566542 4969 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:47.580771 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566544 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566547 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566549 4969 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566552 4969 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566554 4969 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566556 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566559 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566561 4969 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566564 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566568 4969 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566571 4969 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566573 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566576 4969 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566579 4969 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566584 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:47.581340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566606 4969 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.566636 4969 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566670 4969 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566674 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566677 4969 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566682 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566686 4969 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566689 4969 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566692 4969 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566696 4969 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566698 4969 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566701 4969 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566703 4969 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566706 4969 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566709 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566712 4969 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:02:47.581799 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566714 4969 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566717 4969 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566719 4969 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566722 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566724 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566727 4969 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566729 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566732 4969 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566734 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566737 4969 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566739 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566744 4969 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566747 4969 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566749 4969 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566752 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566754 4969 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566757 4969 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566760 4969 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566763 4969 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:02:47.582187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566765 4969 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566768 4969 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566775 4969 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566778 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566780 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566783 4969 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566785 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566788 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566790 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566793 4969 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566795 4969 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566798 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566800 4969 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566803 4969 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566805 4969 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566808 4969 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566810 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566813 4969 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566815 4969 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:02:47.582697 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566818 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566821 4969 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566823 4969 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566826 4969 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566828 4969 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566832 4969 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566835 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566837 4969 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566840 4969 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566842 4969 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566845 4969 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566848 4969 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566850 4969 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566853 4969 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566855 4969 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566863 4969 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566865 4969 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566868 4969 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566870 4969 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566873 4969 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:02:47.583150 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566875 4969 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566878 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566880 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566883 4969 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566885 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566888 4969 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566890 4969 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566893 4969 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566895 4969 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566899 4969 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566902 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566905 4969 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566908 4969 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566910 4969 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566913 4969 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566915 4969 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566918 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566923 4969 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566926 4969 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:02:47.583648 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566928 4969 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566931 4969 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566934 4969 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566936 4969 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566938 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566942 4969 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566944 4969 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566947 4969 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566951 4969 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566953 4969 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566955 4969 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566958 4969 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566960 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566964 4969 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:47.566966 4969 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.567017 4969 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.568016 4969 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.568071 4969 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.568246 4969 server.go:1014] "Starting client certificate rotation" Oct 31 00:02:47.584138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.568395 4969 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:02:47.584613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.568445 4969 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:57:07 +0000 UTC" deadline="2028-05-31 17:55:47.3972159 +0000 UTC" Oct 31 00:02:47.584613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.568494 4969 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13889h52m59.828723728s" Oct 31 00:02:47.584613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.568760 4969 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:02:47.584613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.569899 4969 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:02:47.584613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.571043 4969 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:02:47.584613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.572941 4969 log.go:25] "Validated CRI v1 image API" Oct 31 00:02:47.584613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.573713 4969 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:02:47.584613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.576079 4969 fs.go:135] Filesystem UUIDs: map[1dbcff07-4343-445b-85a1-922f36acb372:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 84003b1f-ab97-4dbe-b9c6-4f080557f62d:/dev/nvme0n1p3] Oct 31 00:02:47.584867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.576098 4969 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00e1004e761e6b65ba3351a61bd00e252e687f111ced112c25ebb34520cac474/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00e1004e761e6b65ba3351a61bd00e252e687f111ced112c25ebb34520cac474/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/051a34b5b8dcbf8a2a176545a793843dce214eec0a6a7d4fe3fcf7a6f2e56213/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/051a34b5b8dcbf8a2a176545a793843dce214eec0a6a7d4fe3fcf7a6f2e56213/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a85d6880acdb0b7fabed86323ac23bb722dd3d6c043659c960da9d2139d1150/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a85d6880acdb0b7fabed86323ac23bb722dd3d6c043659c960da9d2139d1150/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2db816b4acebb162a8a2eebf1591563559e709a1062f95082897f0b8b1f4a8ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2db816b4acebb162a8a2eebf1591563559e709a1062f95082897f0b8b1f4a8ec/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c34b9f4ef3558669226ac92ee841da74687c3dd89abe649c9fae0de2b2194ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c34b9f4ef3558669226ac92ee841da74687c3dd89abe649c9fae0de2b2194ff/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53c5bfff2feb2d51eab3e1af8a0e65353ab452b09fd1ff926cf4f8f31d2ae85b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53c5bfff2feb2d51eab3e1af8a0e65353ab452b09fd1ff926cf4f8f31d2ae85b/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70dadf8838413f441a84e41f6674b8ba2ab7ff3d65f496b26ad4a1343ec04814/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70dadf8838413f441a84e41f6674b8ba2ab7ff3d65f496b26ad4a1343ec04814/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77a3dc24e15b775de804b4b437886ed33b860829ce9add749e3e731d894c95db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77a3dc24e15b775de804b4b437886ed33b860829ce9add749e3e731d894c95db/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a5c74d50b848e85133d216cef218e3a69da9256d9389d6847dbd8814d7f0dbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a5c74d50b848e85133d216cef218e3a69da9256d9389d6847dbd8814d7f0dbb/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8af74d1dca5dbc47f4c725de5bcb8f6b235ea2322baaf9b3897c50c8a9230d8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8af74d1dca5dbc47f4c725de5bcb8f6b235ea2322baaf9b3897c50c8a9230d8c/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8535088ba2febabf26f5b7f1fd42fd50572ebb28f9e1409c2e0227026f969b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8535088ba2febabf26f5b7f1fd42fd50572ebb28f9e1409c2e0227026f969b6/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9cb1e54b0f9b26921211ce2f1ccd1697d7c67dd02a5093387d157afa0ecceb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9cb1e54b0f9b26921211ce2f1ccd1697d7c67dd02a5093387d157afa0ecceb5/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aaf64dda097e48686e67c23d3df1e4e9849995ae9434517fcef5adfc377bf919/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aaf64dda097e48686e67c23d3df1e4e9849995ae9434517fcef5adfc377bf919/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad542f300ec38d49009840ee0f96f7ef5490fda07a8ce28005661e5b2c138c42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad542f300ec38d49009840ee0f96f7ef5490fda07a8ce28005661e5b2c138c42/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd0bab954cc693174b54c365bd57c39dc7f4f1c0b1e120e3c81c9c2dfb5a450c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd0bab954cc693174b54c365bd57c39dc7f4f1c0b1e120e3c81c9c2dfb5a450c/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc88dfe403538dd32c3805c5ca4047a0acea7a98f9ff1efa238d691756205a75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc88dfe403538dd32c3805c5ca4047a0acea7a98f9ff1efa238d691756205a75/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d998baa9860603a93cbd28e6ef782132cbd53a8de34be248f98d3edd3e3e3f92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d998baa9860603a93cbd28e6ef782132cbd53a8de34be248f98d3edd3e3e3f92/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e22c37fc3709bb955af3ba23729184d35fa89c2d8191ae02bd41036659bdfe3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e22c37fc3709bb955af3ba23729184d35fa89c2d8191ae02bd41036659bdfe3e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4deb00504d59bdbe4b5f253d92328842bd7c772a7d173ea8cbbb027ad1eb695/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4deb00504d59bdbe4b5f253d92328842bd7c772a7d173ea8cbbb027ad1eb695/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eabce844e1b364b46cbb3a1632d06bcce6d4cf41987ed8935083f8272c2db223/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eabce844e1b364b46cbb3a1632d06bcce6d4cf41987ed8935083f8272c2db223/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebd1860f367302fd08345faa048cd8261e0cae3ed328f609693d2f483234bb39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebd1860f367302fd08345faa048cd8261e0cae3ed328f609693d2f483234bb39/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec4a891f5c5714f6ef38a1c5785ae98691f80ae88bfad910e8a7f49301d7ed6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec4a891f5c5714f6ef38a1c5785ae98691f80ae88bfad910e8a7f49301d7ed6b/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f332a2a05f016bc90997d92fc76adf1b84e0f7f688ec750805aeef931ea06528/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f332a2a05f016bc90997d92fc76adf1b84e0f7f688ec750805aeef931ea06528/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffff463688490cac01318c689b70f40d4c6c716da109739c5bc77448e9c49513/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffff463688490cac01318c689b70f40d4c6c716da109739c5bc77448e9c49513/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0793ea2b-51e4-48e8-98b3-7cbbec1c410c/volumes/kubernetes.io~projected/kube-api-access-zgvjl:{mountpoint:/var/lib/kubelet/pods/0793ea2b-51e4-48e8-98b3-7cbbec1c410c/volumes/kubernetes.io~projected/kube-api-access-zgvjl major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~projected/kube-api-access-p5xd2:{mountpoint:/var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~projected/kube-api-access-p5xd2 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dc5ea4f-6317-48ca-b075-5a292d72148c/volumes/kubernetes.io~projected/kube-api-access-rr4q2:{mountpoint:/var/lib/kubelet/pods/0dc5ea4f-6317-48ca-b075-5a292d72148c/volumes/kubernetes.io~projected/kube-api-access-rr4q2 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dc5ea4f-6317-48ca-b075-5a292d72148c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0dc5ea4f-6317-48ca-b075-5a292d72148c/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fefd28c-b0d1-4863-819d-2d273ea5e9ef/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0fefd28c-b0d1-4863-819d-2d273ea5e9ef/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/143a4be9-3aa9-4329-88a7-71e995e72ca8/volumes/kubernetes.io~projected/kube-api-access-pn7pk:{mountpoint:/var/lib/kubelet/pods/143a4be9-3aa9-4329-88a7-71e995e72ca8/volumes/kubernetes.io~projected/kube-api-access-pn7pk major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1650ce51-15b5-4454-9c29-2a05cb986307/volumes/kubernetes.io~projected/kube-api-access-dpsr4:{mountpoint:/var/lib/kubelet/pods/1650ce51-15b5-4454-9c29-2a05cb986307/volumes/kubernetes.io~projected/kube-api-access-dpsr4 major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a785fc6-0a05-45d9-875a-6e831047d381/volumes/kubernetes.io~projected/kube-api-access-flcqq:{mountpoint:/var/lib/kubelet/pods/1a785fc6-0a05-45d9-875a-6e831047d381/volumes/kubernetes.io~projected/kube-api-access-flcqq major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d364537-0ba4-4fca-86c0-0575fd61a814/volumes/kubernetes.io~projected/kube-api-access-wnmw9:{mountpoint:/var/lib/kubelet/pods/1d364537-0ba4-4fca-86c0-0575fd61a814/volumes/kubernetes.io~projected/kube-api-access-wnmw9 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b251574-895b-4ac8-9688-3e094921637a/volumes/kubernetes.io~projected/kube-api-access-nhc89:{mountpoint:/var/lib/kubelet/pods/2b251574-895b-4ac8-9688-3e094921637a/volumes/kubernetes.io~projected/kube-api-access-nhc89 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b251574-895b-4ac8-9688-3e094921637a/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/2b251574-895b-4ac8-9688-3e094921637a/volumes/kubernetes.io~secret/default-certificate major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b251574-895b-4ac8-9688-3e094921637a/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/2b251574-895b-4ac8-9688-3e094921637a/volumes/kubernetes.io~secret/stats-auth major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36d4a105-7994-478f-96dc-5e168df0203c/volumes/kubernetes.io~projected/kube-api-access-snd7s:{mountpoint:/var/lib/kubelet/pods/36d4a105-7994-478f-96dc-5e168df0203c/volumes/kubernetes.io~projected/kube-api-access-snd7s major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39778405-de8e-4ae1-8d8f-d2943dc7856b/volumes/kubernetes.io~projected/kube-api-access-tbh75:{mountpoint:/var/lib/kubelet/pods/39778405-de8e-4ae1-8d8f-d2943dc7856b/volumes/kubernetes.io~projected/kube-api-access-tbh75 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39778405-de8e-4ae1-8d8f-d2943dc7856b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/39778405-de8e-4ae1-8d8f-d2943dc7856b/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b6a63f0-3097-4425-b952-e569b827a1d9/volumes/kubernetes.io~projected/kube-api-access-tqxqw:{mountpoint:/var/lib/kubelet/pods/3b6a63f0-3097-4425-b952-e569b827a1d9/volumes/kubernetes.io~projected/kube-api-access-tqxqw major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ec94e45-f0e1-4e95-be72-9201b65344a5/volumes/kubernetes.io~projected/kube-api-access-rmg5n:{mountpoint:/var/lib/kubelet/pods/4ec94e45-f0e1-4e95-be72-9201b65344a5/volumes/kubernetes.io~projected/kube-api-access-rmg5n major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/584395de-91ba-4df6-84c3-0320c0d81cd6/volumes/kubernetes.io~projected/kube-api-access-tnpb7:{mountpoint:/var/lib/kubelet/pods/584395de-91ba-4df6-84c3-0320c0d81cd6/volumes/kubernetes.io~projected/kube-api-access-tnpb7 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69851eab-3b0f-4805-b6fe-2e58e42c21b0/volumes/kubernetes.io~projected/kube-api-access-nqv9c:{mountpoint:/var/lib/kubelet/pods/69851eab-3b0f-4805-b6fe-2e58e42c21b0/volumes/kubernetes.io~projected/kube-api-access-nqv9c major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cb93632-a513-4916-8377-3e2987a32f3d/volumes/kubernetes.io~projected/kube-api-access-msxlv:{mountpoint:/var/lib/kubelet/pods/8cb93632-a513-4916-8377-3e2987a32f3d/volumes/kubernetes.io~projected/kube-api-access-msxlv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bedb70d-0e4f-43e3-98a7-533396bcdf84/volumes/kubernetes.io~projected/kube-api-access-zlrbn:{mountpoint:/var/lib/kubelet/pods/9bedb70d-0e4f-43e3-98a7-533396bcdf84/volumes/kubernetes.io~projected/kube-api-access-zlrbn major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a00d1d31-58a5-4835-ad53-56c65f5adbb0/volumes/kubernetes.io~projected/kube-api-access-7mj2x:{mountpoint:/var/lib/kubelet/pods/a00d1d31-58a5-4835-ad53-56c65f5adbb0/volumes/kubernetes.io~projected/kube-api-access-7mj2x major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a00d1d31-58a5-4835-ad53-56c65f5adbb0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a00d1d31-58a5-4835-ad53-56c65f5adbb0/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0f9a47b-58ae-4467-94a0-fc93927af017/volumes/kubernetes.io~projected/kube-api-access-5ssf7:{mountpoint:/var/lib/kubelet/pods/a0f9a47b-58ae-4467-94a0-fc93927af017/volumes/kubernetes.io~projected/kube-api-access-5ssf7 major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af42676c-2cbc-47a8-a5a9-f38976f43064/volumes/kubernetes.io~projected/kube-api-access-cbwmb:{mountpoint:/var/lib/kubelet/pods/af42676c-2cbc-47a8-a5a9-f38976f43064/volumes/kubernetes.io~projected/kube-api-access-cbwmb major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af42676c-2cbc-47a8-a5a9-f38976f43064/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/af42676c-2cbc-47a8-a5a9-f38976f43064/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af8c82f1-cb80-4dae-a124-df184eab0f73/volumes/kubernetes.io~projected/kube-api-access-7ktj4:{mountpoint:/var/lib/kubelet/pods/af8c82f1-cb80-4dae-a124-df184eab0f73/volumes/kubernetes.io~projected/kube-api-access-7ktj4 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7be51f6-c5b1-4a67-b598-a915f5ff76e8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b7be51f6-c5b1-4a67-b598-a915f5ff76e8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7be51f6-c5b1-4a67-b598-a915f5ff76e8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b7be51f6-c5b1-4a67-b598-a915f5ff76e8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7be51f6-c5b1-4a67-b598-a915f5ff76e8/volumes/kubernetes.io~projected/kube-api-access-qdx2b:{mountpoint:/var/lib/kubelet/pods/b7be51f6-c5b1-4a67-b598-a915f5ff76e8/volumes/kubernetes.io~projected/kube-api-access-qdx2b major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a8a723-d8db-407f-be86-055839588460/volumes/kubernetes.io~projected/kube-api-access-gs7cq:{mountpoint:/var/lib/kubelet/pods/c0a8a723-d8db-407f-be86-055839588460/volumes/kubernetes.io~projected/kube-api-access-gs7cq major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d/volumes/kubernetes.io~projected/kube-api-access-vstjm:{mountpoint:/var/lib/kubelet/pods/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d/volumes/kubernetes.io~projected/kube-api-access-vstjm major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c52f470a-aeaa-44a3-aa2d-7af9aa98e883/volumes/kubernetes.io~projected/kube-api-access-lzsxr:{mountpoint:/var/lib/kubelet/pods/c52f470a-aeaa-44a3-aa2d-7af9aa98e883/volumes/kubernetes.io~projected/kube-api-access-lzsxr major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~projected/kube-api-access-qgft8:{mountpoint:/var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~projected/kube-api-access-qgft8 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9a6cf79-72a1-4c08-ba7d-6b431911167a/volumes/kubernetes.io~projected/kube-api-access-58bmx:{mountpoint:/var/lib/kubelet/pods/d9a6cf79-72a1-4c08-ba7d-6b431911167a/volumes/kubernetes.io~projected/kube-api-access-58bmx major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb324747-4db5-4661-a18e-35933b8ad5ad/volumes/kubernetes.io~projected/kube-api-access-9x2pt:{mountpoint:/var/lib/kubelet/pods/eb324747-4db5-4661-a18e-35933b8ad5ad/volumes/kubernetes.io~projected/kube-api-access-9x2pt major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f61f7ab1-7e07-4f35-9ae8-9e1012e44557/volumes/kubernetes.io~projected/kube-api-access-9flhs:{mountpoint:/var/lib/kubelet/pods/f61f7ab1-7e07-4f35-9ae8-9e1012e44557/volumes/kubernetes.io~projected/kube-api-access-9flhs major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f61f7ab1-7e07-4f35-9ae8-9e1012e44557/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f61f7ab1-7e07-4f35-9ae8-9e1012e44557/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fabccc0d-dbcd-4185-b0dd-1ba72182cc94/volumes/kubernetes.io~projected/kube-api-access-626d2:{mountpoint:/var/lib/kubelet/pods/fabccc0d-dbcd-4185-b0dd-1ba72182cc94/volumes/kubernetes.io~projected/kube-api-access-626d2 major:0 minor:234 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/76f87317302bb04c58ae67357891f30794abc2b383f69dfc5d443d9f74d44981/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7f0fd125f2eac187c14434b7cb457db29be71c59648c33c5993c88e033073e89/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6845a68af40bba85454649fc5fb7046debfd550fd38232f1fa92e74f52ab74ee/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/839f9c52cbf0521258102a1bbde7b23c96d2438a89528340ad75a04a038c4175/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9374f751a516f42f647b3e6b59c26bd7ea7defa233dc8a05c40457e23780125b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/83f6b352ea2d77a5a217b5299caf7414bf38637cc62955eb0289d65be7b249a7/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1018e20e444496a9a18e307be484758a4113e7a0af707d72025f0af81539b438/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6f156c907c16151426b2687b6d9e4bb605e2e6accd838ec66a791cf98c894446/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/eb303b2d1e9ad27cfb9a03d1a9b9724de008948173e8ee889250e04900beeb51/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/4769f6d50c7b3fd6e23300ec04fc9634f2f6250171ee771953f523e856896882/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/acece45f057db00ec79c45fbd0277f1d9536ee1d62c2bf0a6936c943ecab70c9/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/1bf072f673da69afe493c3770b08eef24f551f795be9dfa3c2d0f86bc9714541/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d74e0ddc30471813e8d99ce8be64ee5dcac1fc80876de697c0522ccaa4655d3d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3bec04beb59069faa518d08b22b133660e26058ffde442b9b30ec4d9dff29982/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/85564b5691d7bd133b5e521901e4a18b2b900b2b3008d80f4fe580e2c567c555/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8007ed2b28b31813f0f9a79eaf326ffc63552113e24bc920a734d46e8a751a9e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/318919a3088dcb979de2c27dd927b0f854a206a78c225d4849b0e32dad8eb287/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c422d0523bd2e463cb0f0ec68238e49d33c69ea0240dc1e0b4a9b92c71f4f978/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8aab8ff53f517f3db564c173be011ba7565f9099b7f55d9ba19bdff1477de7fe/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0fe0e14c278133882b7e7d5e5b7cb95424247d304064212d7a61e97bb5ea9819/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/19ca3f374e3c0f661c4b1eebb9f3d3568338a162fdfa5587bbfcd9463006b801/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/62d5f76ba4098199f01b32927927a4e08479ee257d7d67f19e2c8f069ee60c72/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/654859580aa33396c9262755f441d948469e323bd8f13a827110e3cf41735abc/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/b49bf2ef2a60a79999fa642331e8061bbe96a803acb84e66fbe541f81de2df12/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/d025be12b198e7e00809316a06fa83fe8e8ae40e32e6f5e6913e498c1eb60e27/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/825511b656921e6bbe71cc94c62361f19fa732995716743bc711c5d699e04571/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/9faa7f0d8467f3ab8059991fa52d3ca023839acf63e7a957bdc2f23025e7ee6a/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/354f25cf7b15eeb44d994f2b54cf41e350249c494e217afa3e974868dc170e86/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/7bcfbc88146c46105d35ed6d8b00e160457d9a4fa121e96497938807a6846da3/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/5d219cf61e1fbb36e797255eb630264586a156a8f185a8ca38fe93e5bf7f08f6/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/14d1ba34c31250629972e05d63201ce58cec9d538a76a5a2872ddf5686b5ca3d/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/2f5e5f7853b3118fbbd8fa9d2041ebe7196cf81b181169a1de9205077e1bbd0b/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/e071e5fe154160f86c7d8de14b330bc75887d25bc33ccda578073ed8f02ed9ca/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/5b24fe74d7a83bf16c6a43845f386c9c51d24d2f0be5fd94d0ac9882c1a73b38/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/94809cc453a4efa70b7e57f1c1e8d200ad909c3ab683e76d5d5338e933fa7fc0/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/0a7d8247e4b2bdf7a1259baea776a92b46f13f9a0dfde884c80a1f1907ecca88/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/0b9ddf31e662945c7a3154050e9e2433586853173e7b9afb7cd48d16dfb5a4d5/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/745250080c28b36a4c75a0fefe0b90e44acc4041b67a688d84ae3df1535a23a0/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/1bfd926b341f93e6f175aeb3be5daad037b72bc7216725493e3ddcae0724828c/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/71137387eefbdf8c79081911ca6b1c325b0f2b0b9d381129675043dac608c258/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/94a48d9e0270777d4dd235f34debae90db55a7e8d5ba18a0354c9faced335770/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/7cfbfdc68c45c3d6dda36a116980d27016230216d4d34292f815aa006fa51f8a/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/7e34de4409f52a7f85bf3d49fe388d972da946f756579e0a73dbfba576c42222/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/cb2ded4888e18ac64697150c056e49944786f8876f696573fcc9f260479fd0b5/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/68d931575111c38d79923abbcda679eac2392157858043fb32c05caef7c3a4ae/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/da4af619aecf24c25a72d867ce15bc63f413d118dfdb61cace6524ce627b40ba/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/d2b70a60c10f11416e338e142dd08cfa29ac82128ee1ba615f79b55436735c8e/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/1243a9b92a255d3465106aba4fe5d2616d35e09bce1a5d48e038a41ab48adb22/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/929d42d8701a6326717d8ab96c56a1b730769de014003f7c3d9288b5a636e71c/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c7b730e98757a903398f1768b5abc69ec0dbecc73e601d4b122e74696eec5de3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a52a15cf5157dc366efeb7cfa74027748ca547d7f7cc1e13ba324399d18bcce6/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/6041bc16936ed44af55eb33a1ec1593913b6c4875af6e5e37b773e95a66e4515/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/6c28670ac35c6b6a78b73cf36ed4b5ded2944d54b44da35f35b94168c3c68957/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/2a4a8f0cb12b45950a2313b13360f5bfe92fbde63a58fd1f9c89fb668e2dfdda/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3e1e3f5fada9fbfb5d0cd1afe34e2bb3dd71b583071a2c63547afad9a222c49a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5bf7f87e4ee0c26e71db0b794082b3571f88fe31b81e03af80866500a36bc622/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/926e70f8e0420a52b765693b5645afb845311cb4494069161cef470e7f02ddfb/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 00:02:47.585076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.584632 4969 manager.go:217] Machine: {Timestamp:2026-10-31 00:02:47.584107844 +0000 UTC m=+0.059466287 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3179718 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22060b8e021cc96e8eb9a647c85114 SystemUUID:ec22060b-8e02-1cc9-6e8e-b9a647c85114 BootID:d944ecae-aad4-4af6-b3d2-15cb56f6e833 Filesystems:[{Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1650ce51-15b5-4454-9c29-2a05cb986307/volumes/kubernetes.io~projected/kube-api-access-dpsr4 DeviceMajor:0 DeviceMinor:306 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0fefd28c-b0d1-4863-819d-2d273ea5e9ef/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8cb93632-a513-4916-8377-3e2987a32f3d/volumes/kubernetes.io~projected/kube-api-access-msxlv DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a5c74d50b848e85133d216cef218e3a69da9256d9389d6847dbd8814d7f0dbb/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0dc5ea4f-6317-48ca-b075-5a292d72148c/volumes/kubernetes.io~projected/kube-api-access-rr4q2 DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2b251574-895b-4ac8-9688-3e094921637a/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a00d1d31-58a5-4835-ad53-56c65f5adbb0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eabce844e1b364b46cbb3a1632d06bcce6d4cf41987ed8935083f8272c2db223/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f61f7ab1-7e07-4f35-9ae8-9e1012e44557/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f61f7ab1-7e07-4f35-9ae8-9e1012e44557/volumes/kubernetes.io~projected/kube-api-access-9flhs DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8535088ba2febabf26f5b7f1fd42fd50572ebb28f9e1409c2e0227026f969b6/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec4a891f5c5714f6ef38a1c5785ae98691f80ae88bfad910e8a7f49301d7ed6b/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebd1860f367302fd08345faa048cd8261e0cae3ed328f609693d2f483234bb39/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a85d6880acdb0b7fabed86323ac23bb722dd3d6c043659c960da9d2139d1150/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36d4a105-7994-478f-96dc-5e168df0203c/volumes/kubernetes.io~projected/kube-api-access-snd7s DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b251574-895b-4ac8-9688-3e094921637a/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c52f470a-aeaa-44a3-aa2d-7af9aa98e883/volumes/kubernetes.io~projected/kube-api-access-lzsxr DeviceMajor:0 DeviceMinor:238 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53c5bfff2feb2d51eab3e1af8a0e65353ab452b09fd1ff926cf4f8f31d2ae85b/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4deb00504d59bdbe4b5f253d92328842bd7c772a7d173ea8cbbb027ad1eb695/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b7be51f6-c5b1-4a67-b598-a915f5ff76e8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/39778405-de8e-4ae1-8d8f-d2943dc7856b/volumes/kubernetes.io~projected/kube-api-access-tbh75 DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/143a4be9-3aa9-4329-88a7-71e995e72ca8/volumes/kubernetes.io~projected/kube-api-access-pn7pk DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9cb1e54b0f9b26921211ce2f1ccd1697d7c67dd02a5093387d157afa0ecceb5/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/69851eab-3b0f-4805-b6fe-2e58e42c21b0/volumes/kubernetes.io~projected/kube-api-access-nqv9c DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fabccc0d-dbcd-4185-b0dd-1ba72182cc94/volumes/kubernetes.io~projected/kube-api-access-626d2 DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~projected/kube-api-access-qgft8 DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c34b9f4ef3558669226ac92ee841da74687c3dd89abe649c9fae0de2b2194ff/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70dadf8838413f441a84e41f6674b8ba2ab7ff3d65f496b26ad4a1343ec04814/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2db816b4acebb162a8a2eebf1591563559e709a1062f95082897f0b8b1f4a8ec/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0f9a47b-58ae-4467-94a0-fc93927af017/volumes/kubernetes.io~projected/kube-api-access-5ssf7 DeviceMajor:0 DeviceMinor:351 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffff463688490cac01318c689b70f40d4c6c716da109739c5bc77448e9c49513/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e22c37fc3709bb955af3ba23729184d35fa89c2d8191ae02bd41036659bdfe3e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/051a34b5b8dcbf8a2a176545a793843dce214eec0a6a7d4fe3fcf7a6f2e56213/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af42676c-2cbc-47a8-a5a9-f38976f43064/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a00d1d31-58a5-4835-ad53-56c65f5adbb0/volumes/kubernetes.io~projected/kube-api-access-7mj2x DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a8a723-d8db-407f-be86-055839588460/volumes/kubernetes.io~projected/kube-api-access-gs7cq DeviceMajor:0 DeviceMinor:239 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad542f300ec38d49009840ee0f96f7ef5490fda07a8ce28005661e5b2c138c42/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39778405-de8e-4ae1-8d8f-d2943dc7856b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8af74d1dca5dbc47f4c725de5bcb8f6b235ea2322baaf9b3897c50c8a9230d8c/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9a6cf79-72a1-4c08-ba7d-6b431911167a/volumes/kubernetes.io~projected/kube-api-access-58bmx DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77a3dc24e15b775de804b4b437886ed33b860829ce9add749e3e731d894c95db/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d998baa9860603a93cbd28e6ef782132cbd53a8de34be248f98d3edd3e3e3f92/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1d364537-0ba4-4fca-86c0-0575fd61a814/volumes/kubernetes.io~projected/kube-api-access-wnmw9 DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f332a2a05f016bc90997d92fc76adf1b84e0f7f688ec750805aeef931ea06528/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b6a63f0-3097-4425-b952-e569b827a1d9/volumes/kubernetes.io~projected/kube-api-access-tqxqw DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dc5ea4f-6317-48ca-b075-5a292d72148c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af8c82f1-cb80-4dae-a124-df184eab0f73/volumes/kubernetes.io~projected/kube-api-access-7ktj4 DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b7be51f6-c5b1-4a67-b598-a915f5ff76e8/volumes/kubernetes.io~projected/kube-api-access-qdx2b DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/584395de-91ba-4df6-84c3-0320c0d81cd6/volumes/kubernetes.io~projected/kube-api-access-tnpb7 DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af42676c-2cbc-47a8-a5a9-f38976f43064/volumes/kubernetes.io~projected/kube-api-access-cbwmb DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a785fc6-0a05-45d9-875a-6e831047d381/volumes/kubernetes.io~projected/kube-api-access-flcqq DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd0bab954cc693174b54c365bd57c39dc7f4f1c0b1e120e3c81c9c2dfb5a450c/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:299 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d/volumes/kubernetes.io~projected/kube-api-access-vstjm DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aaf64dda097e48686e67c23d3df1e4e9849995ae9434517fcef5adfc377bf919/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eb324747-4db5-4661-a18e-35933b8ad5ad/volumes/kubernetes.io~projected/kube-api-access-9x2pt DeviceMajor:0 DeviceMinor:320 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bedb70d-0e4f-43e3-98a7-533396bcdf84/volumes/kubernetes.io~projected/kube-api-access-zlrbn DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7be51f6-c5b1-4a67-b598-a915f5ff76e8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes/kubernetes.io~projected/kube-api-access-p5xd2 DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b251574-895b-4ac8-9688-3e094921637a/volumes/kubernetes.io~projected/kube-api-access-nhc89 DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00e1004e761e6b65ba3351a61bd00e252e687f111ced112c25ebb34520cac474/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc88dfe403538dd32c3805c5ca4047a0acea7a98f9ff1efa238d691756205a75/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4ec94e45-f0e1-4e95-be72-9201b65344a5/volumes/kubernetes.io~projected/kube-api-access-rmg5n DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0793ea2b-51e4-48e8-98b3-7cbbec1c410c/volumes/kubernetes.io~projected/kube-api-access-zgvjl DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00e1004e761e6b6 MacAddress:72:07:a8:2f:68:4d Speed:10000 Mtu:8901} {Name:051a34b5b8dcbf8 MacAddress:66:75:32:31:7a:88 Speed:10000 Mtu:8901} {Name:53c5bfff2feb2d5 MacAddress:ee:13:e9:83:bd:11 Speed:10000 Mtu:8901} {Name:70dadf8838413f4 MacAddress:d6:fb:29:64:06:08 Speed:10000 Mtu:8901} {Name:8a5c74d50b848e8 MacAddress:da:b6:38:bd:04:72 Speed:10000 Mtu:8901} {Name:8af74d1dca5dbc4 MacAddress:86:05:f0:fb:26:10 Speed:10000 Mtu:8901} {Name:a8535088ba2feba MacAddress:f2:5e:88:ee:49:e9 Speed:10000 Mtu:8901} {Name:aaf64dda097e486 MacAddress:ba:ca:cb:53:ea:05 Speed:10000 Mtu:8901} {Name:bd0bab954cc6931 MacAddress:26:cd:f3:d4:4c:f8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:48:ad:7b:de:97 Speed:0 Mtu:9001} {Name:br-int MacAddress:16:cd:84:a9:c6:d5 Speed:0 Mtu:8901} {Name:cc88dfe403538dd MacAddress:12:54:93:33:ed:e0 Speed:10000 Mtu:8901} {Name:e4deb00504d59bd MacAddress:c2:26:5e:81:b5:85 Speed:10000 Mtu:8901} {Name:eabce844e1b364b MacAddress:0e:d0:d0:fe:da:71 Speed:10000 Mtu:8901} {Name:ec4a891f5c5714f MacAddress:f2:1a:ca:0f:67:1e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:48:ad:7b:de:97 Speed:0 Mtu:9001} {Name:f332a2a05f016bc MacAddress:5a:be:53:dc:9e:aa Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c6:31:d3:f8:3b:82 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:8f:63:eb:31:a5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:02:47.585076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585016 4969 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:02:47.585076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585062 4969 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:02:47.585470 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585412 4969 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:02:47.585696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585474 4969 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-63.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:02:47.585696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585658 4969 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:02:47.585696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585669 4969 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:02:47.585696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585683 4969 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:02:47.585912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585701 4969 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:02:47.585912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585727 4969 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:02:47.586002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.585933 4969 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:02:47.586074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.586062 4969 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:02:47.586121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.586110 4969 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:02:47.586173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.586125 4969 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:02:47.586173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.586140 4969 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:02:47.586173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.586155 4969 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:02:47.586173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.586165 4969 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:02:47.586863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.586847 4969 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:02:47.586863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.586862 4969 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:02:47.587363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.587348 4969 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:02:47.688932 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.688913 4969 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:02:47.688932 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.688930 4969 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:02:47.689045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689027 4969 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:02:47.689227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689215 4969 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:02:47.689268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689246 4969 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:02:47.689362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689353 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:02:47.689397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689370 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:02:47.689397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689378 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:02:47.689397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689386 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:02:47.689397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689395 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:02:47.689512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689404 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:02:47.689512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689413 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:02:47.689512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689420 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:02:47.689512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689427 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:02:47.689512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689433 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:02:47.689512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689452 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:02:47.689512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689462 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:02:47.689512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689492 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:02:47.689512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.689500 4969 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:02:47.693150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.693135 4969 server.go:1263] "Started kubelet" Oct 31 00:02:47.693264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.693224 4969 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:02:47.697079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.693823 4969 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:02:47.697079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.694143 4969 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:02:47.697079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.694432 4969 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:02:47.697079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.695833 4969 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:02:47.694366 ip-10-0-141-63 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:02:47.699771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.699753 4969 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.699906 4969 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:47.699916 4969 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-63.us-west-1.compute.internal\" not found" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.699946 4969 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:57:08 +0000 UTC" deadline="2028-07-27 02:08:41.5690546 +0000 UTC" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.699962 4969 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15242h5m53.869095759s" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.699968 4969 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.699976 4969 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.700055 4969 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.700725 4969 factory.go:55] Registering systemd factory Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.700740 4969 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.701143 4969 factory.go:153] Registering CRI-O factory Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.701157 4969 factory.go:223] Registration of the crio container factory successfully Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.701209 4969 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.701230 4969 factory.go:103] Registering Raw factory Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.701248 4969 manager.go:1199] Started watching for new ooms in manager Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.701621 4969 manager.go:319] Starting recovery of all containers Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.702870 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3d78a36-3b13-449d-bfb9-b25cb3e9d12d" volumeName="kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-env-overrides" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.702922 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9a6cf79-72a1-4c08-ba7d-6b431911167a" volumeName="kubernetes.io/projected/d9a6cf79-72a1-4c08-ba7d-6b431911167a-kube-api-access-58bmx" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.702937 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fabccc0d-dbcd-4185-b0dd-1ba72182cc94" volumeName="kubernetes.io/configmap/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-telemetry-config" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.702950 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0793ea2b-51e4-48e8-98b3-7cbbec1c410c" volumeName="kubernetes.io/configmap/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.702963 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36d4a105-7994-478f-96dc-5e168df0203c" volumeName="kubernetes.io/empty-dir/36d4a105-7994-478f-96dc-5e168df0203c-tmp-dir" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.702976 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b6a63f0-3097-4425-b952-e569b827a1d9" volumeName="kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.702988 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ec94e45-f0e1-4e95-be72-9201b65344a5" volumeName="kubernetes.io/configmap/4ec94e45-f0e1-4e95-be72-9201b65344a5-serviceca" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703000 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fabccc0d-dbcd-4185-b0dd-1ba72182cc94" volumeName="kubernetes.io/projected/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-kube-api-access-626d2" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703015 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a785fc6-0a05-45d9-875a-6e831047d381" volumeName="kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-cni-binary-copy" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703029 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b251574-895b-4ac8-9688-3e094921637a" volumeName="kubernetes.io/projected/2b251574-895b-4ac8-9688-3e094921637a-kube-api-access-nhc89" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703043 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc5ea4f-6317-48ca-b075-5a292d72148c" volumeName="kubernetes.io/projected/0dc5ea4f-6317-48ca-b075-5a292d72148c-kube-api-access-rr4q2" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703054 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a00d1d31-58a5-4835-ad53-56c65f5adbb0" volumeName="kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-trusted-ca" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703065 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c52f470a-aeaa-44a3-aa2d-7af9aa98e883" volumeName="kubernetes.io/configmap/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-config-volume" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703083 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f61f7ab1-7e07-4f35-9ae8-9e1012e44557" volumeName="kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-snapshots" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703100 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f5b61-0433-4481-9c79-064d608b8300" volumeName="kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703113 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bedb70d-0e4f-43e3-98a7-533396bcdf84" volumeName="kubernetes.io/projected/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kube-api-access-zlrbn" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703126 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f61f7ab1-7e07-4f35-9ae8-9e1012e44557" volumeName="kubernetes.io/secret/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-serving-cert" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703137 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7be51f6-c5b1-4a67-b598-a915f5ff76e8" volumeName="kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-tmp" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703151 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a8a723-d8db-407f-be86-055839588460" volumeName="kubernetes.io/projected/c0a8a723-d8db-407f-be86-055839588460-kube-api-access-gs7cq" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703163 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc5ea4f-6317-48ca-b075-5a292d72148c" volumeName="kubernetes.io/secret/0dc5ea4f-6317-48ca-b075-5a292d72148c-serving-cert" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703174 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b6a63f0-3097-4425-b952-e569b827a1d9" volumeName="kubernetes.io/projected/3b6a63f0-3097-4425-b952-e569b827a1d9-kube-api-access-tqxqw" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703202 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cb93632-a513-4916-8377-3e2987a32f3d" volumeName="kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703215 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a00d1d31-58a5-4835-ad53-56c65f5adbb0" volumeName="kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-config" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703227 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46dedb6-cf0a-489e-9094-75e60189c87f" volumeName="kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703240 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f61f7ab1-7e07-4f35-9ae8-9e1012e44557" volumeName="kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-service-ca-bundle" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703252 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b251574-895b-4ac8-9688-3e094921637a" volumeName="kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-default-certificate" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703280 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af42676c-2cbc-47a8-a5a9-f38976f43064" volumeName="kubernetes.io/configmap/af42676c-2cbc-47a8-a5a9-f38976f43064-config" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703294 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af42676c-2cbc-47a8-a5a9-f38976f43064" volumeName="kubernetes.io/projected/af42676c-2cbc-47a8-a5a9-f38976f43064-kube-api-access-cbwmb" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703312 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3d78a36-3b13-449d-bfb9-b25cb3e9d12d" volumeName="kubernetes.io/secret/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703324 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f5b61-0433-4481-9c79-064d608b8300" volumeName="kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703337 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f5b61-0433-4481-9c79-064d608b8300" volumeName="kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703349 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cb93632-a513-4916-8377-3e2987a32f3d" volumeName="kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703360 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af42676c-2cbc-47a8-a5a9-f38976f43064" volumeName="kubernetes.io/secret/af42676c-2cbc-47a8-a5a9-f38976f43064-serving-cert" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703372 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7be51f6-c5b1-4a67-b598-a915f5ff76e8" volumeName="kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-tuned" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703386 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f5b61-0433-4481-9c79-064d608b8300" volumeName="kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703408 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a785fc6-0a05-45d9-875a-6e831047d381" volumeName="kubernetes.io/projected/1a785fc6-0a05-45d9-875a-6e831047d381-kube-api-access-flcqq" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703419 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb324747-4db5-4661-a18e-35933b8ad5ad" volumeName="kubernetes.io/projected/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-api-access-9x2pt" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703433 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3d78a36-3b13-449d-bfb9-b25cb3e9d12d" volumeName="kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703447 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3d78a36-3b13-449d-bfb9-b25cb3e9d12d" volumeName="kubernetes.io/projected/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-kube-api-access-vstjm" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703458 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0793ea2b-51e4-48e8-98b3-7cbbec1c410c" volumeName="kubernetes.io/projected/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-kube-api-access-zgvjl" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703470 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc5ea4f-6317-48ca-b075-5a292d72148c" volumeName="kubernetes.io/configmap/0dc5ea4f-6317-48ca-b075-5a292d72148c-config" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703482 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b6a63f0-3097-4425-b952-e569b827a1d9" volumeName="kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703495 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ec94e45-f0e1-4e95-be72-9201b65344a5" volumeName="kubernetes.io/projected/4ec94e45-f0e1-4e95-be72-9201b65344a5-kube-api-access-rmg5n" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703506 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0f9a47b-58ae-4467-94a0-fc93927af017" volumeName="kubernetes.io/projected/a0f9a47b-58ae-4467-94a0-fc93927af017-kube-api-access-5ssf7" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703517 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c52f470a-aeaa-44a3-aa2d-7af9aa98e883" volumeName="kubernetes.io/projected/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-kube-api-access-lzsxr" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703529 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46dedb6-cf0a-489e-9094-75e60189c87f" volumeName="kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703542 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46dedb6-cf0a-489e-9094-75e60189c87f" volumeName="kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703555 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fefd28c-b0d1-4863-819d-2d273ea5e9ef" volumeName="kubernetes.io/configmap/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-konnectivity-ca" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703567 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="143a4be9-3aa9-4329-88a7-71e995e72ca8" volumeName="kubernetes.io/projected/143a4be9-3aa9-4329-88a7-71e995e72ca8-kube-api-access-pn7pk" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703579 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39778405-de8e-4ae1-8d8f-d2943dc7856b" volumeName="kubernetes.io/projected/39778405-de8e-4ae1-8d8f-d2943dc7856b-kube-api-access-tbh75" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703621 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46dedb6-cf0a-489e-9094-75e60189c87f" volumeName="kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703634 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46dedb6-cf0a-489e-9094-75e60189c87f" volumeName="kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703660 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9a6cf79-72a1-4c08-ba7d-6b431911167a" volumeName="kubernetes.io/configmap/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-config" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703673 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb324747-4db5-4661-a18e-35933b8ad5ad" volumeName="kubernetes.io/empty-dir/eb324747-4db5-4661-a18e-35933b8ad5ad-data-volume" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703688 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f61f7ab1-7e07-4f35-9ae8-9e1012e44557" volumeName="kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703700 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f5b61-0433-4481-9c79-064d608b8300" volumeName="kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703720 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fefd28c-b0d1-4863-819d-2d273ea5e9ef" volumeName="kubernetes.io/secret/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-agent-certs" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703733 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b251574-895b-4ac8-9688-3e094921637a" volumeName="kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-stats-auth" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703746 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69851eab-3b0f-4805-b6fe-2e58e42c21b0" volumeName="kubernetes.io/projected/69851eab-3b0f-4805-b6fe-2e58e42c21b0-kube-api-access-nqv9c" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703762 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a00d1d31-58a5-4835-ad53-56c65f5adbb0" volumeName="kubernetes.io/secret/a00d1d31-58a5-4835-ad53-56c65f5adbb0-serving-cert" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703775 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f61f7ab1-7e07-4f35-9ae8-9e1012e44557" volumeName="kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-tmp" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703787 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a785fc6-0a05-45d9-875a-6e831047d381" volumeName="kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-multus-daemon-config" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703800 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d364537-0ba4-4fca-86c0-0575fd61a814" volumeName="kubernetes.io/projected/1d364537-0ba4-4fca-86c0-0575fd61a814-kube-api-access-wnmw9" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703812 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32201f7b-9d56-4eb1-a3f4-b344b88beee9" volumeName="kubernetes.io/configmap/32201f7b-9d56-4eb1-a3f4-b344b88beee9-nginx-conf" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703825 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39778405-de8e-4ae1-8d8f-d2943dc7856b" volumeName="kubernetes.io/secret/39778405-de8e-4ae1-8d8f-d2943dc7856b-cert" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703836 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b6a63f0-3097-4425-b952-e569b827a1d9" volumeName="kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-binary-copy" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703850 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="584395de-91ba-4df6-84c3-0320c0d81cd6" volumeName="kubernetes.io/projected/584395de-91ba-4df6-84c3-0320c0d81cd6-kube-api-access-tnpb7" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703862 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454" volumeName="kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703874 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f5b61-0433-4481-9c79-064d608b8300" volumeName="kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703887 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a00d1d31-58a5-4835-ad53-56c65f5adbb0" volumeName="kubernetes.io/projected/a00d1d31-58a5-4835-ad53-56c65f5adbb0-kube-api-access-7mj2x" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703900 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af8c82f1-cb80-4dae-a124-df184eab0f73" volumeName="kubernetes.io/projected/af8c82f1-cb80-4dae-a124-df184eab0f73-kube-api-access-7ktj4" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703916 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c52f470a-aeaa-44a3-aa2d-7af9aa98e883" volumeName="kubernetes.io/empty-dir/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-tmp-dir" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703927 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46dedb6-cf0a-489e-9094-75e60189c87f" volumeName="kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703939 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d46dedb6-cf0a-489e-9094-75e60189c87f" volumeName="kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703951 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f61f7ab1-7e07-4f35-9ae8-9e1012e44557" volumeName="kubernetes.io/projected/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-kube-api-access-9flhs" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703964 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f5b61-0433-4481-9c79-064d608b8300" volumeName="kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703977 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1650ce51-15b5-4454-9c29-2a05cb986307" volumeName="kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.703989 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36d4a105-7994-478f-96dc-5e168df0203c" volumeName="kubernetes.io/projected/36d4a105-7994-478f-96dc-5e168df0203c-kube-api-access-snd7s" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.704001 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7be51f6-c5b1-4a67-b598-a915f5ff76e8" volumeName="kubernetes.io/projected/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-kube-api-access-qdx2b" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.704014 4969 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3d78a36-3b13-449d-bfb9-b25cb3e9d12d" volumeName="kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-config" seLinuxMountContext="" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.704027 4969 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:02:47.704627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.704034 4969 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:02:47.708969 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.708947 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/1.log" Oct 31 00:02:47.713327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.713169 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/1.log" Oct 31 00:02:47.724746 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.724721 4969 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:02:47.726169 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.726146 4969 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:02:47.726278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.726176 4969 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:02:47.726278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.726199 4969 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:02:47.726278 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:47.726238 4969 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:02:47.741191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.741163 4969 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="8a2c22bda96c5ac654d80f5625ee74518e1a0a14ce316b382455b494dd656916" exitCode=0 Oct 31 00:02:47.741264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.741193 4969 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="4a4900f4f31f42165ceaed42fd981aacfd992d2ea8b5e1c100417f8a1d33b845" exitCode=0 Oct 31 00:02:47.741264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.741203 4969 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="a734dde24fde72b75da059f3f90bb1350691cb056e9cf60a96972f900a00b1f8" exitCode=0 Oct 31 00:02:47.741264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.741211 4969 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="ab70a876d71a54c2254f97dcfc78e8bf2cf031960898504dbff826346d5fa932" exitCode=0 Oct 31 00:02:47.741264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.741218 4969 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="37249db7afaa496b8f1769c827d3f40f1a0189309e5530ae24d08934c347e763" exitCode=0 Oct 31 00:02:47.741264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.741226 4969 generic.go:356] "Generic (PLEG): container finished" podID="3b6a63f0-3097-4425-b952-e569b827a1d9" containerID="bdc221653c5ce31e84a06b05b20ded22d2d8d2ce854b6cd731742363dd864c7a" exitCode=0 Oct 31 00:02:47.743139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.743116 4969 generic.go:356] "Generic (PLEG): container finished" podID="53f30427e3252533f78f5cc535c34236" containerID="58925b14a9fc2d5d46859193802a9073b261eb6cfa1bf19f9741a005431c1d58" exitCode=0 Oct 31 00:02:47.749327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.749309 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/1.log" Oct 31 00:02:47.749399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.749338 4969 generic.go:356] "Generic (PLEG): container finished" podID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" containerID="fc82fc114325880ddef36554f3c84dca7816e34a56beb8db5c5fc2e991a794dd" exitCode=255 Oct 31 00:02:47.792497 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.792474 4969 manager.go:324] Recovery completed Oct 31 00:02:47.797351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.797328 4969 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:47.811508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811492 4969 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:02:47.811508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811505 4969 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:02:47.811644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811520 4969 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:02:47.811675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811659 4969 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:02:47.811719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811671 4969 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:02:47.811765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811721 4969 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:02:47.811765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811726 4969 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:02:47.811765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811731 4969 policy_none.go:50] "Start" Oct 31 00:02:47.811765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811750 4969 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:02:47.811765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811760 4969 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:02:47.811906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811898 4969 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:02:47.811936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811907 4969 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:02:47.811936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.811923 4969 policy_none.go:44] "Start" Oct 31 00:02:47.817624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.817582 4969 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:02:47.822940 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.822927 4969 manager.go:341] "Starting Device Plugin manager" Oct 31 00:02:47.823019 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:47.822951 4969 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:02:47.823019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.822960 4969 server.go:87] "Starting device plugin registration server" Oct 31 00:02:47.823220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.823206 4969 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:02:47.823274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.823223 4969 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:02:47.823330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.823314 4969 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:02:47.823466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.823418 4969 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:02:47.823466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.823464 4969 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:02:47.823579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.823472 4969 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:02:47.824099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.823501 4969 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:02:47.8234533Z","UUID":"12aa8737-69e4-4bd2-9ca0-09eb7a72570d","Handler":null,"Name":"","Endpoint":""} Oct 31 00:02:47.824791 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:47.824770 4969 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:02:47.825212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.825187 4969 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:02:47.825284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.825224 4969 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:02:47.826336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.826293 4969 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal"] Oct 31 00:02:47.827005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.826969 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"53f30427e3252533f78f5cc535c34236","Type":"ContainerStarted","Data":"1d72c23711f29fba923302406a4cc9fb4ce216f3b438c8ac00502e7d49a6e419"} Oct 31 00:02:47.827005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.826999 4969 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:47.827158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.827015 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"53f30427e3252533f78f5cc535c34236","Type":"ContainerDied","Data":"58925b14a9fc2d5d46859193802a9073b261eb6cfa1bf19f9741a005431c1d58"} Oct 31 00:02:47.827158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.827035 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"53f30427e3252533f78f5cc535c34236","Type":"ContainerStarted","Data":"ebd1860f367302fd08345faa048cd8261e0cae3ed328f609693d2f483234bb39"} Oct 31 00:02:47.827158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.827136 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"020b2f1cd0c673bc8a3307af2e07b1c3","Type":"ContainerStarted","Data":"9cc3a7ae20bbcf7d225c38514448b9d81aea3691e513b7d703555f5bb4319298"} Oct 31 00:02:47.827314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.827158 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" event={"ID":"020b2f1cd0c673bc8a3307af2e07b1c3","Type":"ContainerStarted","Data":"ffff463688490cac01318c689b70f40d4c6c716da109739c5bc77448e9c49513"} Oct 31 00:02:47.827314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.827198 4969 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:47.847838 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.847811 4969 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:47.847999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.847975 4969 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:47.848182 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.848164 4969 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:02:47.848244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.848195 4969 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:02:47.887483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.887452 4969 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:02:47.887642 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:47.887519 4969 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:47.895932 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.895913 4969 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:02:47.896050 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:47.895964 4969 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:47.924308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.924286 4969 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:02:47.925715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.925697 4969 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:02:47.925819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.925743 4969 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:02:47.925819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.925769 4969 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-63.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:02:47.925913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.925835 4969 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:47.948995 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.948939 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:47.949394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.948985 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:47.949614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:47.949565 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/020b2f1cd0c673bc8a3307af2e07b1c3-config\") pod \"kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"020b2f1cd0c673bc8a3307af2e07b1c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.037870 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.037840 4969 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.038009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.037913 4969 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.050686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.050662 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.050811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.050691 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.050811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.050710 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/020b2f1cd0c673bc8a3307af2e07b1c3-config\") pod \"kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"020b2f1cd0c673bc8a3307af2e07b1c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.050811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.050756 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/020b2f1cd0c673bc8a3307af2e07b1c3-config\") pod \"kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"020b2f1cd0c673bc8a3307af2e07b1c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.050811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.050780 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.050958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.050781 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53f30427e3252533f78f5cc535c34236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" (UID: \"53f30427e3252533f78f5cc535c34236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.586540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.586466 4969 apiserver.go:52] "Watching apiserver" Oct 31 00:02:48.739535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.739490 4969 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:02:48.740227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.740202 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-9h7rh","openshift-ingress-canary/ingress-canary-62mlb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb","openshift-image-registry/image-pruner-29890080-867tw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl","kube-system/kas-connection-checker-685895c7c9-rgl6b","openshift-image-registry/node-ca-r5nkn","kube-system/konnectivity-agent-5hrcq","openshift-dns/dns-default-2hhtz","openshift-image-registry/image-registry-869c89c69d-stxfd","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4","openshift-insights/insights-runtime-extractor-hzjx9","openshift-multus/multus-8rpzc","openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4","kube-system/kas-connection-checker-685895c7c9-czswt","openshift-cluster-node-tuning-operator/tuned-fzxls","openshift-ovn-kubernetes/ovnkube-node-4b5st","openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd","kube-system/kas-connection-checker-685895c7c9-fzds7","openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj","openshift-insights/insights-operator-7c4986d449-7rj9b","openshift-console-operator/console-operator-7cd56b8dbc-wqpvv","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c","openshift-network-operator/iptables-alerter-9c62n","openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw","openshift-image-registry/image-registry-794c4fd7c9-8n2f4","openshift-ingress/router-default-94bcddf7f-vbt59","openshift-multus/multus-additional-cni-plugins-s4t9z","openshift-multus/network-metrics-daemon-mgbdr","kube-system/global-pull-secret-syncer-24nkz","openshift-network-console/networking-console-plugin-86c478bb88-9wfdj","openshift-network-diagnostics/network-check-target-qgstv"] Oct 31 00:02:48.741585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.741565 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:48.741918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.741899 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.742507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.742487 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:48.743773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.743756 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.744786 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.744769 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:48.750252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.750231 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:48.753864 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.753842 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.754956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.754941 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:48.755460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.755445 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.780913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.780891 4969 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.781059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.781046 4969 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.793317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793292 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.793406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793331 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.793406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793341 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 00:02:48.793406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793369 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.793406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793375 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 00:02:48.793406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793400 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.793620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793443 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.793676 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793664 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 00:02:48.793720 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793692 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 00:02:48.793946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793934 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 00:02:48.793977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793948 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 00:02:48.793977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.793950 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.794233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.794221 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.794311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.794247 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 00:02:48.794350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.794322 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.794769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.794758 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 00:02:48.794805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.794772 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.795110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.795099 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.801749 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.801736 4969 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:02:48.851163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.851112 4969 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:02:48.851163 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.851157 4969 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.854555 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854537 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrbn\" (UniqueName: \"kubernetes.io/projected/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kube-api-access-zlrbn\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.854619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854564 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-k8s-cni-cncf-io\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.854619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854583 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flhs\" (UniqueName: \"kubernetes.io/projected/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-kube-api-access-9flhs\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.854715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854639 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpb7\" (UniqueName: \"kubernetes.io/projected/584395de-91ba-4df6-84c3-0320c0d81cd6-kube-api-access-tnpb7\") pod \"kas-connection-checker-685895c7c9-rgl6b\" (UID: \"584395de-91ba-4df6-84c3-0320c0d81cd6\") " pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" Oct 31 00:02:48.854715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854665 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snd7s\" (UniqueName: \"kubernetes.io/projected/36d4a105-7994-478f-96dc-5e168df0203c-kube-api-access-snd7s\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:48.854839 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854812 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-host\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.854880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854862 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb324747-4db5-4661-a18e-35933b8ad5ad-data-volume\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.854919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854889 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-dbus\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:48.854959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854905 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-kubelet\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.854959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854938 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb324747-4db5-4661-a18e-35933b8ad5ad-data-volume\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.854959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854952 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-systemd-units\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.855068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.854983 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:48.855537 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855047 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.855616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855566 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/32201f7b-9d56-4eb1-a3f4-b344b88beee9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:48.855670 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855618 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-snapshots\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.855670 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855649 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.855775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855677 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-config\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.855775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855702 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0dc5ea4f-6317-48ca-b075-5a292d72148c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:48.855775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855712 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-snapshots\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.855775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855731 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:48.855775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855758 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-multus-certs\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.855944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855796 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-env-overrides\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.855944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855826 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ssf7\" (UniqueName: \"kubernetes.io/projected/a0f9a47b-58ae-4467-94a0-fc93927af017-kube-api-access-5ssf7\") pod \"migrator-66df5d496b-hsrfj\" (UID: \"a0f9a47b-58ae-4467-94a0-fc93927af017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj" Oct 31 00:02:48.855944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855864 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbwmb\" (UniqueName: \"kubernetes.io/projected/af42676c-2cbc-47a8-a5a9-f38976f43064-kube-api-access-cbwmb\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:48.855944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855895 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0dc5ea4f-6317-48ca-b075-5a292d72148c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:48.855944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855895 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-modprobe-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.855944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855929 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-sys-fs\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855968 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-conf-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.855984 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-kubernetes\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856020 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-run\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856040 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-multus-daemon-config\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856067 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-etc-kubernetes\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856090 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ec94e45-f0e1-4e95-be72-9201b65344a5-host\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856111 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr4q2\" (UniqueName: \"kubernetes.io/projected/0dc5ea4f-6317-48ca-b075-5a292d72148c-kube-api-access-rr4q2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856129 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856145 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-systemd\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856169 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstjm\" (UniqueName: \"kubernetes.io/projected/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-kube-api-access-vstjm\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.856184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856184 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb324747-4db5-4661-a18e-35933b8ad5ad-crio-socket\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856204 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856296 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a00d1d31-58a5-4835-ad53-56c65f5adbb0-serving-cert\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856326 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36d4a105-7994-478f-96dc-5e168df0203c-hosts-file\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856351 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-lib-modules\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856356 4969 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856377 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovn-node-metrics-cert\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.856396 4969 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-63.us-west-1.compute.internal" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856403 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856428 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d4a105-7994-478f-96dc-5e168df0203c-tmp-dir\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856481 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856544 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856572 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc89\" (UniqueName: \"kubernetes.io/projected/2b251574-895b-4ac8-9688-3e094921637a-kube-api-access-nhc89\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856580 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d4a105-7994-478f-96dc-5e168df0203c-tmp-dir\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:48.856612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856614 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxlv\" (UniqueName: \"kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856648 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-tmp\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856673 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbh75\" (UniqueName: \"kubernetes.io/projected/39778405-de8e-4ae1-8d8f-d2943dc7856b-kube-api-access-tbh75\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856700 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0dc5ea4f-6317-48ca-b075-5a292d72148c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856721 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-tmp\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856820 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn7pk\" (UniqueName: \"kubernetes.io/projected/143a4be9-3aa9-4329-88a7-71e995e72ca8-kube-api-access-pn7pk\") pod \"kas-connection-checker-685895c7c9-fzds7\" (UID: \"143a4be9-3aa9-4329-88a7-71e995e72ca8\") " pod="kube-system/kas-connection-checker-685895c7c9-fzds7" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856828 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0dc5ea4f-6317-48ca-b075-5a292d72148c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856849 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856878 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856906 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-konnectivity-ca\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856970 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvjl\" (UniqueName: \"kubernetes.io/projected/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-kube-api-access-zgvjl\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.856996 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mj2x\" (UniqueName: \"kubernetes.io/projected/a00d1d31-58a5-4835-ad53-56c65f5adbb0-kube-api-access-7mj2x\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857021 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-etc-selinux\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857110 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-log-socket\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.857142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857136 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-system-cni-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857154 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcqq\" (UniqueName: \"kubernetes.io/projected/1a785fc6-0a05-45d9-875a-6e831047d381-kube-api-access-flcqq\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857170 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqv9c\" (UniqueName: \"kubernetes.io/projected/69851eab-3b0f-4805-b6fe-2e58e42c21b0-kube-api-access-nqv9c\") pod \"kas-connection-checker-685895c7c9-czswt\" (UID: \"69851eab-3b0f-4805-b6fe-2e58e42c21b0\") " pod="kube-system/kas-connection-checker-685895c7c9-czswt" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857194 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857247 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-etc-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857279 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysconfig\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857298 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857349 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-stats-auth\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857380 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857448 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-tmp-dir\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857468 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-sys\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.857524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857511 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-slash\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.857849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857551 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-tmp-dir\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:48.857849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857554 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxqw\" (UniqueName: \"kubernetes.io/projected/3b6a63f0-3097-4425-b952-e569b827a1d9-kube-api-access-tqxqw\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.857849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857627 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmg5n\" (UniqueName: \"kubernetes.io/projected/4ec94e45-f0e1-4e95-be72-9201b65344a5-kube-api-access-rmg5n\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:48.857849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857715 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-trusted-ca\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:48.857849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857735 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:48.857849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857747 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-stats-auth\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.858051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857859 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-ovn\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.858051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857887 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.858051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857915 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.858051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857951 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.858051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857972 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:48.858051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.857989 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:48.858051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858008 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:48.858051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858030 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.858367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858133 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.858367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858164 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-kubelet-config\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:48.858367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858191 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-hostroot\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.858367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858222 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-service-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.858367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858243 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-var-lib-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.858367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858266 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:48.858367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858302 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-cnibin\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.858367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858322 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39778405-de8e-4ae1-8d8f-d2943dc7856b-cert\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:48.858575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858391 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626d2\" (UniqueName: \"kubernetes.io/projected/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-kube-api-access-626d2\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:48.858575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858415 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.858575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858438 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-conf\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.858575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858462 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-netd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.858575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858491 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzsxr\" (UniqueName: \"kubernetes.io/projected/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-kube-api-access-lzsxr\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:48.858575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858562 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnmw9\" (UniqueName: \"kubernetes.io/projected/1d364537-0ba4-4fca-86c0-0575fd61a814-kube-api-access-wnmw9\") pod \"network-check-source-6f78d59f99-hq2vd\" (UID: \"1d364537-0ba4-4fca-86c0-0575fd61a814\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" Oct 31 00:02:48.858756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858640 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-registration-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.858756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858690 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-bin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.858756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858713 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2pt\" (UniqueName: \"kubernetes.io/projected/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-api-access-9x2pt\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.858756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858734 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/af8c82f1-cb80-4dae-a124-df184eab0f73-kube-api-access-7ktj4\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:48.858869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858757 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-cnibin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.858869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858790 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.858946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858919 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:48.858995 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858958 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgft8\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.859051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.858991 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5xd2\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.859051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859020 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-tuned\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.859139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859050 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.859139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859077 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.859139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859104 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.859139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859109 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-tuned\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.859266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859140 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-system-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.859266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859195 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-netns\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.859266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859224 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af42676c-2cbc-47a8-a5a9-f38976f43064-config\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:48.859266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859246 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-bin\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.859447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859268 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-script-lib\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.859447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859350 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7cq\" (UniqueName: \"kubernetes.io/projected/c0a8a723-d8db-407f-be86-055839588460-kube-api-access-gs7cq\") pod \"volume-data-source-validator-58c9f5f46d-nddrl\" (UID: \"c0a8a723-d8db-407f-be86-055839588460\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" Oct 31 00:02:48.859447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859383 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.859447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859412 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-socket-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.859447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859439 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.859685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859452 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.859685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859465 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.859685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859505 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-iptables-alerter-script\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:48.859685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859524 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-config-volume\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:48.859685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859544 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.859685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859580 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.859685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859632 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-multus\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.859685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859654 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-default-certificate\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.859685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859670 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ec94e45-f0e1-4e95-be72-9201b65344a5-serviceca\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859688 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859717 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-host-slash\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859743 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-config\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859789 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859813 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859831 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-default-certificate\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859832 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bmx\" (UniqueName: \"kubernetes.io/projected/d9a6cf79-72a1-4c08-ba7d-6b431911167a-kube-api-access-58bmx\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859861 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af42676c-2cbc-47a8-a5a9-f38976f43064-serving-cert\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859885 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859885 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-socket-dir-parent\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859921 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.859985 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860013 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-var-lib-kubelet\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860037 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-tmp\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.860095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860073 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-device-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.860558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860131 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-systemd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.860558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860148 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-tmp\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.860558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860214 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-cni-binary-copy\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.860558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860242 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-serving-cert\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.860558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860263 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-node-log\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.860558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860283 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdx2b\" (UniqueName: \"kubernetes.io/projected/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-kube-api-access-qdx2b\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.860558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860309 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-netns\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.860558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860343 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-kubelet\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.860558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.860367 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-agent-certs\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:48.894020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.893997 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.894787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.894766 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.896269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.896244 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:02:48.896425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.896312 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 00:02:48.897115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.897091 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.899673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899651 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.899811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899672 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af42676c-2cbc-47a8-a5a9-f38976f43064-config\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:48.899811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899659 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 00:02:48.899811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899672 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.899811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899727 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.899811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899731 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:02:48.899811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899765 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 00:02:48.899811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899777 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899823 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899866 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xx4g5\"" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899881 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899913 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899952 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.899971 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ec94e45-f0e1-4e95-be72-9201b65344a5-serviceca\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900002 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900077 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900082 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:02:48.900129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900080 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900186 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900204 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gvgd9\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900349 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqqqs\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900366 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5ddpr\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900390 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900398 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900436 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900461 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900487 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pp2sk\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900499 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900529 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900531 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 00:02:48.900606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900567 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900632 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hplth\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900676 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900710 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900715 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900764 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900773 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900861 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-agent-certs\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.900923 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901028 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901044 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901081 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-27h5h\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901092 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901110 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901122 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901030 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901097 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901183 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.901206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901198 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.901773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901221 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:02:48.901773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901329 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:02:48.901773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901397 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.901773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901464 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 00:02:48.901773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901494 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.901773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.901556 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.902030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902015 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x2zk\"" Oct 31 00:02:48.902078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902054 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:02:48.902124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902075 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zm4vk\"" Oct 31 00:02:48.902124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902078 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.902124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902104 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4gwp\"" Oct 31 00:02:48.902124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902094 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.902124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902124 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zgjmv\"" Oct 31 00:02:48.902278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902139 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fsk29\"" Oct 31 00:02:48.902278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902143 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 00:02:48.902278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902183 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-pc4hm\"" Oct 31 00:02:48.902278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902259 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7xfz6\"" Oct 31 00:02:48.902419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902302 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:02:48.902419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902330 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.902419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902353 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.902419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902356 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:48.902419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902389 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wr2gb\"" Oct 31 00:02:48.902582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902418 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:02:48.902582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902446 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:02:48.902582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.902451 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:02:48.905904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.905885 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-config\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.905980 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.905888 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/32201f7b-9d56-4eb1-a3f4-b344b88beee9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:48.905980 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.905974 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-env-overrides\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.907218 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.907200 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.907290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.907214 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovn-node-metrics-cert\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.907290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.907235 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.907290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.907246 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a00d1d31-58a5-4835-ad53-56c65f5adbb0-serving-cert\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:48.907389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.907289 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fefd28c-b0d1-4863-819d-2d273ea5e9ef-konnectivity-ca\") pod \"konnectivity-agent-5hrcq\" (UID: \"0fefd28c-b0d1-4863-819d-2d273ea5e9ef\") " pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:48.907389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.907358 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-multus-daemon-config\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.908464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.908444 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:48.908533 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.908444 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-service-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.908580 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.908541 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39778405-de8e-4ae1-8d8f-d2943dc7856b-cert\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:48.909707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.909686 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.909773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.909716 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-config-volume\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:48.909773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.909731 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.909885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.909782 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.909885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.909798 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-iptables-alerter-script\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:48.909885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.909806 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.909991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.909885 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b6a63f0-3097-4425-b952-e569b827a1d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.909991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.909941 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-config\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:48.910065 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.910017 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af42676c-2cbc-47a8-a5a9-f38976f43064-serving-cert\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:48.910852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.910838 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-serving-cert\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.910977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.910954 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a785fc6-0a05-45d9-875a-6e831047d381-cni-binary-copy\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.947231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.947198 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:02:48.947340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.947244 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jt67z\"" Oct 31 00:02:48.947340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.947246 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 00:02:48.947340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.947261 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z67qm\"" Oct 31 00:02:48.947340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.947274 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7lc6\"" Oct 31 00:02:48.947340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.947292 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtt85\"" Oct 31 00:02:48.947526 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.947394 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:02:48.947609 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.947576 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 00:02:48.948191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.948168 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 00:02:48.948460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.948436 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-original-pull-secret\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:48.948521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.948506 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d1d31-58a5-4835-ad53-56c65f5adbb0-trusted-ca\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:48.949102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949075 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.949102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949087 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpb7\" (UniqueName: \"kubernetes.io/projected/584395de-91ba-4df6-84c3-0320c0d81cd6-kube-api-access-tnpb7\") pod \"kas-connection-checker-685895c7c9-rgl6b\" (UID: \"584395de-91ba-4df6-84c3-0320c0d81cd6\") " pod="kube-system/kas-connection-checker-685895c7c9-rgl6b" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949170 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrbn\" (UniqueName: \"kubernetes.io/projected/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kube-api-access-zlrbn\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949265 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbwmb\" (UniqueName: \"kubernetes.io/projected/af42676c-2cbc-47a8-a5a9-f38976f43064-kube-api-access-cbwmb\") pod \"service-ca-operator-544d749c8c-2gwk4\" (UID: \"af42676c-2cbc-47a8-a5a9-f38976f43064\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949334 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snd7s\" (UniqueName: \"kubernetes.io/projected/36d4a105-7994-478f-96dc-5e168df0203c-kube-api-access-snd7s\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949361 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flhs\" (UniqueName: \"kubernetes.io/projected/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-kube-api-access-9flhs\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949565 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstjm\" (UniqueName: \"kubernetes.io/projected/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-kube-api-access-vstjm\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949647 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvjl\" (UniqueName: \"kubernetes.io/projected/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-kube-api-access-zgvjl\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949704 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-ovnkube-script-lib\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949743 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr4q2\" (UniqueName: \"kubernetes.io/projected/0dc5ea4f-6317-48ca-b075-5a292d72148c-kube-api-access-rr4q2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c\" (UID: \"0dc5ea4f-6317-48ca-b075-5a292d72148c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.949924 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn7pk\" (UniqueName: \"kubernetes.io/projected/143a4be9-3aa9-4329-88a7-71e995e72ca8-kube-api-access-pn7pk\") pod \"kas-connection-checker-685895c7c9-fzds7\" (UID: \"143a4be9-3aa9-4329-88a7-71e995e72ca8\") " pod="kube-system/kas-connection-checker-685895c7c9-fzds7" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950091 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbh75\" (UniqueName: \"kubernetes.io/projected/39778405-de8e-4ae1-8d8f-d2943dc7856b-kube-api-access-tbh75\") pod \"ingress-canary-62mlb\" (UID: \"39778405-de8e-4ae1-8d8f-d2943dc7856b\") " pod="openshift-ingress-canary/ingress-canary-62mlb" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950118 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxlv\" (UniqueName: \"kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv\") pod \"image-pruner-29890080-867tw\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950168 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdx2b\" (UniqueName: \"kubernetes.io/projected/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-kube-api-access-qdx2b\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950174 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/af8c82f1-cb80-4dae-a124-df184eab0f73-kube-api-access-7ktj4\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950184 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950292 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcqq\" (UniqueName: \"kubernetes.io/projected/1a785fc6-0a05-45d9-875a-6e831047d381-kube-api-access-flcqq\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950481 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bmx\" (UniqueName: \"kubernetes.io/projected/d9a6cf79-72a1-4c08-ba7d-6b431911167a-kube-api-access-58bmx\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950656 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqv9c\" (UniqueName: \"kubernetes.io/projected/69851eab-3b0f-4805-b6fe-2e58e42c21b0-kube-api-access-nqv9c\") pod \"kas-connection-checker-685895c7c9-czswt\" (UID: \"69851eab-3b0f-4805-b6fe-2e58e42c21b0\") " pod="kube-system/kas-connection-checker-685895c7c9-czswt" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950671 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzsxr\" (UniqueName: \"kubernetes.io/projected/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-kube-api-access-lzsxr\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950661 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpsr4\" (UniqueName: \"kubernetes.io/projected/1650ce51-15b5-4454-9c29-2a05cb986307-kube-api-access-dpsr4\") pod \"network-check-target-qgstv\" (UID: \"1650ce51-15b5-4454-9c29-2a05cb986307\") " pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950706 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmg5n\" (UniqueName: \"kubernetes.io/projected/4ec94e45-f0e1-4e95-be72-9201b65344a5-kube-api-access-rmg5n\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950714 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ssf7\" (UniqueName: \"kubernetes.io/projected/a0f9a47b-58ae-4467-94a0-fc93927af017-kube-api-access-5ssf7\") pod \"migrator-66df5d496b-hsrfj\" (UID: \"a0f9a47b-58ae-4467-94a0-fc93927af017\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj" Oct 31 00:02:48.950766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950734 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7cq\" (UniqueName: \"kubernetes.io/projected/c0a8a723-d8db-407f-be86-055839588460-kube-api-access-gs7cq\") pod \"volume-data-source-validator-58c9f5f46d-nddrl\" (UID: \"c0a8a723-d8db-407f-be86-055839588460\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-nddrl" Oct 31 00:02:48.951665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950861 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5xd2\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.951665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.950922 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhc89\" (UniqueName: \"kubernetes.io/projected/2b251574-895b-4ac8-9688-3e094921637a-kube-api-access-nhc89\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.951665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.951296 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxqw\" (UniqueName: \"kubernetes.io/projected/3b6a63f0-3097-4425-b952-e569b827a1d9-kube-api-access-tqxqw\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.951665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.951369 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.951665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.951413 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mj2x\" (UniqueName: \"kubernetes.io/projected/a00d1d31-58a5-4835-ad53-56c65f5adbb0-kube-api-access-7mj2x\") pod \"console-operator-7cd56b8dbc-wqpvv\" (UID: \"a00d1d31-58a5-4835-ad53-56c65f5adbb0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:48.951665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.951424 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgft8\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.951665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.951477 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2pt\" (UniqueName: \"kubernetes.io/projected/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-api-access-9x2pt\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.951665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.951552 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626d2\" (UniqueName: \"kubernetes.io/projected/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-kube-api-access-626d2\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:48.951665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.951557 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnmw9\" (UniqueName: \"kubernetes.io/projected/1d364537-0ba4-4fca-86c0-0575fd61a814-kube-api-access-wnmw9\") pod \"network-check-source-6f78d59f99-hq2vd\" (UID: \"1d364537-0ba4-4fca-86c0-0575fd61a814\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hq2vd" Oct 31 00:02:48.957075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.957058 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:48.958235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.958219 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f61f7ab1-7e07-4f35-9ae8-9e1012e44557-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7rj9b\" (UID: \"f61f7ab1-7e07-4f35-9ae8-9e1012e44557\") " pod="openshift-insights/insights-operator-7c4986d449-7rj9b" Oct 31 00:02:48.958482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.958467 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.960453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960436 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.960698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960685 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-ovn\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.960732 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960706 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.960732 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960722 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:48.960799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960737 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:48.960799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960770 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.960866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960799 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-ovn\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.960866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960808 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-kubelet-config\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:48.960866 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.960830 4969 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:48.960866 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.960843 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:48.960866 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.960856 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:48.960866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960866 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-hostroot\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960847 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-kubelet-config\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960841 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-hostroot\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960838 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.960912 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.460891485 +0000 UTC m=+1.936249907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.960934 4969 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.960951 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.460940316 +0000 UTC m=+1.936298718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.960967 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-var-lib-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.960986 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.460975621 +0000 UTC m=+1.936334022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : configmap "kube-rbac-proxy" not found Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961003 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-var-lib-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961009 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-cnibin\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.961047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961041 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961055 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-conf\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961072 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-netd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961095 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-registration-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961115 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-bin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961127 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-cnibin\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961134 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-netd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961136 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-cnibin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961158 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-bin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961170 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961176 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-cnibin\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961184 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961170 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-registration-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961203 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysctl-conf\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961199 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961238 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961244 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.961267 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.461253316 +0000 UTC m=+1.936611719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : configmap references non-existent config key: service-ca.crt Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961287 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-system-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.961293 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.961301 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961304 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-netns\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961321 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-bin\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.961326 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.461317627 +0000 UTC m=+1.936676029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961351 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-socket-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961356 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-system-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961364 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-netns\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961373 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961356 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-cni-bin\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961398 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-multus\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961434 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-cni-multus\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961445 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961446 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-socket-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961466 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-host-slash\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:48.961480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961485 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-socket-dir-parent\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961514 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-socket-dir-parent\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961517 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0793ea2b-51e4-48e8-98b3-7cbbec1c410c-host-slash\") pod \"iptables-alerter-9c62n\" (UID: \"0793ea2b-51e4-48e8-98b3-7cbbec1c410c\") " pod="openshift-network-operator/iptables-alerter-9c62n" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961539 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961556 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-var-lib-kubelet\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961575 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-device-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961626 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-systemd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961630 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-var-lib-kubelet\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961633 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961655 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-systemd\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961663 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-device-dir\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961669 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-node-log\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961709 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-netns\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961734 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-kubelet\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961754 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-node-log\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961760 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-k8s-cni-cncf-io\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961765 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-netns\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961785 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-host\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961793 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-var-lib-kubelet\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961805 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-k8s-cni-cncf-io\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961809 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-dbus\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961833 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-host\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961834 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-kubelet\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961857 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-systemd-units\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961887 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961896 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454-dbus\") pod \"global-pull-secret-syncer-24nkz\" (UID: \"c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454\") " pod="kube-system/global-pull-secret-syncer-24nkz" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961912 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-kubelet\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961915 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-systemd-units\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961935 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961966 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.961977 4969 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961993 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-multus-certs\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.961995 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-run-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962004 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.461995989 +0000 UTC m=+1.937354389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : secret "metrics-daemon-secret" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962028 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-host-run-multus-certs\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962039 4969 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962040 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-modprobe-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962081 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.462072022 +0000 UTC m=+1.937430424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962099 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-sys-fs\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962116 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-conf-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962129 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-modprobe-d\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962132 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-kubernetes\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962147 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-sys-fs\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962151 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-conf-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962166 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-run\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962193 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-etc-kubernetes\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962161 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-kubernetes\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962214 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ec94e45-f0e1-4e95-be72-9201b65344a5-host\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962242 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-etc-kubernetes\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962246 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-run\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962241 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-systemd\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962254 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ec94e45-f0e1-4e95-be72-9201b65344a5-host\") pod \"node-ca-r5nkn\" (UID: \"4ec94e45-f0e1-4e95-be72-9201b65344a5\") " pod="openshift-image-registry/node-ca-r5nkn" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962272 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb324747-4db5-4661-a18e-35933b8ad5ad-crio-socket\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962280 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-systemd\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962288 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962304 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36d4a105-7994-478f-96dc-5e168df0203c-hosts-file\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962317 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-lib-modules\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962322 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb324747-4db5-4661-a18e-35933b8ad5ad-crio-socket\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962331 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962350 4969 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962356 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962373 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.462366436 +0000 UTC m=+1.937724870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : secret "insights-runtime-extractor-tls" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962384 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-etc-selinux\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962391 4969 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962400 4969 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962401 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-log-socket\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962415 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-lib-modules\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962420 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-system-cni-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962425 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls podName:d9a6cf79-72a1-4c08-ba7d-6b431911167a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.462415588 +0000 UTC m=+1.937773990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4fbn4" (UID: "d9a6cf79-72a1-4c08-ba7d-6b431911167a") : secret "samples-operator-tls" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962449 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36d4a105-7994-478f-96dc-5e168df0203c-hosts-file\") pod \"node-resolver-9h7rh\" (UID: \"36d4a105-7994-478f-96dc-5e168df0203c\") " pod="openshift-dns/node-resolver-9h7rh" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962458 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b6a63f0-3097-4425-b952-e569b827a1d9-system-cni-dir\") pod \"multus-additional-cni-plugins-s4t9z\" (UID: \"3b6a63f0-3097-4425-b952-e569b827a1d9\") " pod="openshift-multus/multus-additional-cni-plugins-s4t9z" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962464 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls podName:c52f470a-aeaa-44a3-aa2d-7af9aa98e883 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.462451192 +0000 UTC m=+1.937809592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls") pod "dns-default-2hhtz" (UID: "c52f470a-aeaa-44a3-aa2d-7af9aa98e883") : secret "dns-default-metrics-tls" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962469 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9bedb70d-0e4f-43e3-98a7-533396bcdf84-etc-selinux\") pod \"aws-ebs-csi-driver-node-nbxjb\" (UID: \"9bedb70d-0e4f-43e3-98a7-533396bcdf84\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nbxjb" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962458 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-log-socket\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962502 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962530 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-etc-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962548 4969 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962553 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysconfig\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962577 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:48.962585 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.462576761 +0000 UTC m=+1.937935162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : secret "router-metrics-certs-default" not found Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962621 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-sys\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962626 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-etc-openvswitch\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962637 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-slash\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962646 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-etc-sysconfig\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962652 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7be51f6-c5b1-4a67-b598-a915f5ff76e8-sys\") pod \"tuned-fzxls\" (UID: \"b7be51f6-c5b1-4a67-b598-a915f5ff76e8\") " pod="openshift-cluster-node-tuning-operator/tuned-fzxls" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962648 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a785fc6-0a05-45d9-875a-6e831047d381-multus-cni-dir\") pod \"multus-8rpzc\" (UID: \"1a785fc6-0a05-45d9-875a-6e831047d381\") " pod="openshift-multus/multus-8rpzc" Oct 31 00:02:48.962855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:48.962694 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3d78a36-3b13-449d-bfb9-b25cb3e9d12d-host-slash\") pod \"ovnkube-node-4b5st\" (UID: \"c3d78a36-3b13-449d-bfb9-b25cb3e9d12d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:49.043681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.043661 4969 scope.go:122] "RemoveContainer" containerID="fc82fc114325880ddef36554f3c84dca7816e34a56beb8db5c5fc2e991a794dd" Oct 31 00:02:49.057522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.057502 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:02:49.404540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.404476 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:49.468166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468134 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:49.468166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468167 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:49.468384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468189 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:49.468384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468208 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:49.468384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468283 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:49.468384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468328 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:49.468384 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468297 4969 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:02:49.468384 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468345 4969 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:02:49.468384 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468353 4969 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:02:49.468384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468357 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:49.468384 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468302 4969 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468402 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468382574 +0000 UTC m=+2.943740976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : secret "router-metrics-certs-default" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468319 4969 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468435 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468425868 +0000 UTC m=+2.943784270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : secret "insights-runtime-extractor-tls" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468438 4969 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468455 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468447097 +0000 UTC m=+2.943805500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : configmap "kube-rbac-proxy" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468483 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls podName:c52f470a-aeaa-44a3-aa2d-7af9aa98e883 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468465311 +0000 UTC m=+2.943823726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls") pod "dns-default-2hhtz" (UID: "c52f470a-aeaa-44a3-aa2d-7af9aa98e883") : secret "dns-default-metrics-tls" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468506 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls podName:d9a6cf79-72a1-4c08-ba7d-6b431911167a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468496212 +0000 UTC m=+2.943854614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4fbn4" (UID: "d9a6cf79-72a1-4c08-ba7d-6b431911167a") : secret "samples-operator-tls" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468517 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468523 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468515109 +0000 UTC m=+2.943873514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468531 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468565 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468554695 +0000 UTC m=+2.943913100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468564 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468614 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468667 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:49.468692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468683 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468670936 +0000 UTC m=+2.944029340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : configmap references non-existent config key: service-ca.crt Oct 31 00:02:49.469135 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468721 4969 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:02:49.469135 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.468722 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:49.469135 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468752 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468742518 +0000 UTC m=+2.944100943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : secret "metrics-daemon-secret" not found Oct 31 00:02:49.469135 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468762 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:49.469135 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468773 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:49.469135 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468805 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468789014 +0000 UTC m=+2.944147415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:49.469135 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468816 4969 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:49.469135 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.468842 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:50.468834257 +0000 UTC m=+2.944192659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:49.784726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.784659 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/2.log" Oct 31 00:02:49.785097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.785076 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/1.log" Oct 31 00:02:49.785134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.785103 4969 generic.go:356] "Generic (PLEG): container finished" podID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" containerID="c232d385a18d29902ce4eb1872336791c06708d17fc48dd793f13235e41574cb" exitCode=255 Oct 31 00:02:49.785229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.785209 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" event={"ID":"a00d1d31-58a5-4835-ad53-56c65f5adbb0","Type":"ContainerDied","Data":"c232d385a18d29902ce4eb1872336791c06708d17fc48dd793f13235e41574cb"} Oct 31 00:02:49.785282 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.785271 4969 scope.go:122] "RemoveContainer" containerID="fc82fc114325880ddef36554f3c84dca7816e34a56beb8db5c5fc2e991a794dd" Oct 31 00:02:49.785990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:49.785974 4969 scope.go:122] "RemoveContainer" containerID="c232d385a18d29902ce4eb1872336791c06708d17fc48dd793f13235e41574cb" Oct 31 00:02:49.786146 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:49.786122 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:02:50.477735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.477701 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:50.477735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.477738 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.477763 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.477780 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.477806 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.477836 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.477858 4969 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.477886 4969 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.477895 4969 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.477858 4969 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.477915 4969 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.477930 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.477911436 +0000 UTC m=+4.953269850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : secret "metrics-daemon-secret" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.477869 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.477955 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.477936964 +0000 UTC m=+4.953295384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : secret "insights-runtime-extractor-tls" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.477979 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls podName:c52f470a-aeaa-44a3-aa2d-7af9aa98e883 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.477970486 +0000 UTC m=+4.953328900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls") pod "dns-default-2hhtz" (UID: "c52f470a-aeaa-44a3-aa2d-7af9aa98e883") : secret "dns-default-metrics-tls" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.477987 4969 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478000 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls podName:d9a6cf79-72a1-4c08-ba7d-6b431911167a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.477994963 +0000 UTC m=+4.953353364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4fbn4" (UID: "d9a6cf79-72a1-4c08-ba7d-6b431911167a") : secret "samples-operator-tls" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478012 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.478007524 +0000 UTC m=+4.953365925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.478032 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:50.478042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478042 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.478026696 +0000 UTC m=+4.953385099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : secret "router-metrics-certs-default" not found Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.478081 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478101 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478113 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.478131 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478144 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.478135159 +0000 UTC m=+4.953493582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.478176 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478205 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.47819386 +0000 UTC m=+4.953552292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : configmap references non-existent config key: service-ca.crt Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478206 4969 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478246 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.478235048 +0000 UTC m=+4.953593463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478263 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478277 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.478326 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.478317254 +0000 UTC m=+4.953675656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:50.478827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.478416 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb324747-4db5-4661-a18e-35933b8ad5ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:50.789893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.789826 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/2.log" Oct 31 00:02:50.791384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.791361 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hsrfj" event={"ID":"a0f9a47b-58ae-4467-94a0-fc93927af017","Type":"ContainerStarted","Data":"c64b553dce6bef48d24b3a507d7c83a3a809b36fe26c5059b85919227d751b4e"} Oct 31 00:02:50.950682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.950651 4969 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:50.950932 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:50.950919 4969 scope.go:122] "RemoveContainer" containerID="c232d385a18d29902ce4eb1872336791c06708d17fc48dd793f13235e41574cb" Oct 31 00:02:50.951089 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:50.951073 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:02:52.098682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.098654 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:02:52.099130 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.098925 4969 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:02:52.497241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497199 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:52.497241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497248 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497269 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497288 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497317 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497374 4969 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497388 4969 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497419 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497443 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.497430736 +0000 UTC m=+8.972789137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : secret "router-metrics-certs-default" not found Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497378 4969 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497374 4969 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497476 4969 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497391 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:52.497489 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497498 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497497 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.497474156 +0000 UTC m=+8.972832561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : secret "insights-runtime-extractor-tls" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497523 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls podName:c52f470a-aeaa-44a3-aa2d-7af9aa98e883 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.497515392 +0000 UTC m=+8.972873793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls") pod "dns-default-2hhtz" (UID: "c52f470a-aeaa-44a3-aa2d-7af9aa98e883") : secret "dns-default-metrics-tls" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497534 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls podName:d9a6cf79-72a1-4c08-ba7d-6b431911167a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.497529634 +0000 UTC m=+8.972888034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4fbn4" (UID: "d9a6cf79-72a1-4c08-ba7d-6b431911167a") : secret "samples-operator-tls" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497551 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.497546422 +0000 UTC m=+8.972904822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497566 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.497560979 +0000 UTC m=+8.972919399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497625 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497649 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497690 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.497711 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497749 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497762 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497770 4969 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497781 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle podName:2b251574-895b-4ac8-9688-3e094921637a nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.497762394 +0000 UTC m=+8.973120826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle") pod "router-default-94bcddf7f-vbt59" (UID: "2b251574-895b-4ac8-9688-3e094921637a") : configmap references non-existent config key: service-ca.crt Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497785 4969 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497801 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.497790556 +0000 UTC m=+8.973148995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : secret "metrics-daemon-secret" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497820 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.49781455 +0000 UTC m=+8.973172951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:52.497920 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:52.497831 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:56.497825339 +0000 UTC m=+8.973183739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:52.857827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:52.857761 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9h7rh_36d4a105-7994-478f-96dc-5e168df0203c/dns-node-resolver/0.log" Oct 31 00:02:53.258882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:53.258855 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29890080-867tw_8cb93632-a513-4916-8377-3e2987a32f3d/image-pruner/0.log" Oct 31 00:02:54.058457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:54.058430 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r5nkn_4ec94e45-f0e1-4e95-be72-9201b65344a5/node-ca/0.log" Oct 31 00:02:54.674901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:54.674875 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62mlb_39778405-de8e-4ae1-8d8f-d2943dc7856b/serve-healthcheck-canary/0.log" Oct 31 00:02:54.859813 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:54.859786 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:02:54.860138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:54.860126 4969 scope.go:122] "RemoveContainer" containerID="c232d385a18d29902ce4eb1872336791c06708d17fc48dd793f13235e41574cb" Oct 31 00:02:54.860323 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:54.860305 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:02:55.259161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:55.259131 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-hsrfj_a0f9a47b-58ae-4467-94a0-fc93927af017/migrator/0.log" Oct 31 00:02:55.466666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:55.466633 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c_0dc5ea4f-6317-48ca-b075-5a292d72148c/kube-storage-version-migrator-operator/0.log" Oct 31 00:02:56.537224 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537190 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537226 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537269 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537298 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537356 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537365 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537372 4969 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537387 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794c4fd7c9-8n2f4: secret "image-registry-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537388 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537423 4969 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537428 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537437 4969 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869c89c69d-stxfd: secret "image-registry-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537458 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert podName:32201f7b-9d56-4eb1-a3f4-b344b88beee9 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:04.537441786 +0000 UTC m=+17.012800192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9wfdj" (UID: "32201f7b-9d56-4eb1-a3f4-b344b88beee9") : secret "networking-console-plugin-cert" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537498 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls podName:0b2f5b61-0433-4481-9c79-064d608b8300 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:04.537479411 +0000 UTC m=+17.012837813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls") pod "image-registry-794c4fd7c9-8n2f4" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300") : secret "image-registry-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537509 4969 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537518 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls podName:d46dedb6-cf0a-489e-9094-75e60189c87f nodeName:}" failed. No retries permitted until 2026-10-31 00:03:04.537508692 +0000 UTC m=+17.012867096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls") pod "image-registry-869c89c69d-stxfd" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f") : secret "image-registry-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537524 4969 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537541 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs podName:af8c82f1-cb80-4dae-a124-df184eab0f73 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:04.537531932 +0000 UTC m=+17.012890333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs") pod "network-metrics-daemon-mgbdr" (UID: "af8c82f1-cb80-4dae-a124-df184eab0f73") : secret "metrics-daemon-secret" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537545 4969 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537562 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537570 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls podName:eb324747-4db5-4661-a18e-35933b8ad5ad nodeName:}" failed. No retries permitted until 2026-10-31 00:03:04.537560093 +0000 UTC m=+17.012918508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzjx9" (UID: "eb324747-4db5-4661-a18e-35933b8ad5ad") : secret "insights-runtime-extractor-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537625 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:02:56.537635 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls podName:fabccc0d-dbcd-4185-b0dd-1ba72182cc94 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:04.537619507 +0000 UTC m=+17.012977912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sn7fw" (UID: "fabccc0d-dbcd-4185-b0dd-1ba72182cc94") : secret "cluster-monitoring-operator-tls" not found Oct 31 00:02:56.537702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537674 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:56.538623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537918 4969 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:02:56.538623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.537982 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b251574-895b-4ac8-9688-3e094921637a-service-ca-bundle\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:56.540135 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.540112 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c52f470a-aeaa-44a3-aa2d-7af9aa98e883-metrics-tls\") pod \"dns-default-2hhtz\" (UID: \"c52f470a-aeaa-44a3-aa2d-7af9aa98e883\") " pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:56.540423 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.540403 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9a6cf79-72a1-4c08-ba7d-6b431911167a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4fbn4\" (UID: \"d9a6cf79-72a1-4c08-ba7d-6b431911167a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:56.540461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.540376 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b251574-895b-4ac8-9688-3e094921637a-metrics-certs\") pod \"router-default-94bcddf7f-vbt59\" (UID: \"2b251574-895b-4ac8-9688-3e094921637a\") " pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:56.542196 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.542181 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" Oct 31 00:02:56.542297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.542284 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:56.556191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:56.556150 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:57.019136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:57.019039 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4"] Oct 31 00:02:57.022089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:57.022064 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hhtz"] Oct 31 00:02:57.060877 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:57.060853 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52f470a_aeaa_44a3_aa2d_7af9aa98e883.slice/crio-22d1694d4a8f3faa98429dd5aa9f1a3ab93a3708904b4081fe07d9ca543bc643 WatchSource:0}: Error finding container 22d1694d4a8f3faa98429dd5aa9f1a3ab93a3708904b4081fe07d9ca543bc643: Status 404 returned error can't find the container with id 22d1694d4a8f3faa98429dd5aa9f1a3ab93a3708904b4081fe07d9ca543bc643 Oct 31 00:02:57.068861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:57.068839 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-94bcddf7f-vbt59"] Oct 31 00:02:57.071553 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:02:57.071534 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b251574_895b_4ac8_9688_3e094921637a.slice/crio-3a112f0a48f0b5bcd58f44ef0714dcb46bb0efca4b1513880ea53ed1e9611c7e WatchSource:0}: Error finding container 3a112f0a48f0b5bcd58f44ef0714dcb46bb0efca4b1513880ea53ed1e9611c7e: Status 404 returned error can't find the container with id 3a112f0a48f0b5bcd58f44ef0714dcb46bb0efca4b1513880ea53ed1e9611c7e Oct 31 00:02:57.443903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:57.443864 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:02:57.815561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:57.815479 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hhtz" event={"ID":"c52f470a-aeaa-44a3-aa2d-7af9aa98e883","Type":"ContainerStarted","Data":"22d1694d4a8f3faa98429dd5aa9f1a3ab93a3708904b4081fe07d9ca543bc643"} Oct 31 00:02:57.816739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:57.816717 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" event={"ID":"d9a6cf79-72a1-4c08-ba7d-6b431911167a","Type":"ContainerStarted","Data":"61470771dce8c8cdb77b873345897170d502e11e53e5566059f93bc6da7fb30c"} Oct 31 00:02:57.818125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:57.818107 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-94bcddf7f-vbt59" event={"ID":"2b251574-895b-4ac8-9688-3e094921637a","Type":"ContainerStarted","Data":"b70346b6fe1be13fc1d2168004fba081c284cb193498c69038b3fc4e30f8ddc9"} Oct 31 00:02:57.818215 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:57.818137 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-94bcddf7f-vbt59" event={"ID":"2b251574-895b-4ac8-9688-3e094921637a","Type":"ContainerStarted","Data":"3a112f0a48f0b5bcd58f44ef0714dcb46bb0efca4b1513880ea53ed1e9611c7e"} Oct 31 00:02:58.239622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:58.239576 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5hrcq" Oct 31 00:02:58.542824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:58.542737 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:58.546268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:58.546240 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:58.822161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:58.822083 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:58.823888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:58.823864 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-94bcddf7f-vbt59" Oct 31 00:02:59.827084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:59.827051 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hhtz" event={"ID":"c52f470a-aeaa-44a3-aa2d-7af9aa98e883","Type":"ContainerStarted","Data":"5ccec7e79d36cdc09990890f9d14244889539bdb4d4135548bde6b0ef0f5deef"} Oct 31 00:02:59.827561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:59.827108 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hhtz" event={"ID":"c52f470a-aeaa-44a3-aa2d-7af9aa98e883","Type":"ContainerStarted","Data":"1a6dd5289c1c6cbb8a01891e1a9bc90785ea4bb6bedcda627aab9feffbcf3310"} Oct 31 00:02:59.827561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:59.827163 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2hhtz" Oct 31 00:02:59.829489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:59.829458 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" event={"ID":"d9a6cf79-72a1-4c08-ba7d-6b431911167a","Type":"ContainerStarted","Data":"d2c7ed3ff3b6673713104b9e9c784803e0534ef8d9f8a990de367a51f5f34ffa"} Oct 31 00:02:59.829580 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:02:59.829511 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4fbn4" event={"ID":"d9a6cf79-72a1-4c08-ba7d-6b431911167a","Type":"ContainerStarted","Data":"1b3d775af06722b644f4b21bf8718669bc94c51326207edc69dea8a4db90df70"} Oct 31 00:03:00.727312 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:00.727283 4969 scope.go:122] "RemoveContainer" containerID="c232d385a18d29902ce4eb1872336791c06708d17fc48dd793f13235e41574cb" Oct 31 00:03:00.839297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:00.839274 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:03:00.839722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:00.839706 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/2.log" Oct 31 00:03:00.839873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:00.839841 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" event={"ID":"a00d1d31-58a5-4835-ad53-56c65f5adbb0","Type":"ContainerStarted","Data":"58528d26db0a15e451276149469bf9a6299eeebb5657f4c0921f095970233876"} Oct 31 00:03:00.840534 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:00.840513 4969 scope.go:122] "RemoveContainer" containerID="58528d26db0a15e451276149469bf9a6299eeebb5657f4c0921f095970233876" Oct 31 00:03:00.840688 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:03:00.840670 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:03:00.951175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:00.951148 4969 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:03:01.845255 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:01.845226 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:03:01.845692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:01.845677 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/2.log" Oct 31 00:03:01.845742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:01.845710 4969 generic.go:356] "Generic (PLEG): container finished" podID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" containerID="58528d26db0a15e451276149469bf9a6299eeebb5657f4c0921f095970233876" exitCode=255 Oct 31 00:03:01.845780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:01.845755 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" event={"ID":"a00d1d31-58a5-4835-ad53-56c65f5adbb0","Type":"ContainerDied","Data":"58528d26db0a15e451276149469bf9a6299eeebb5657f4c0921f095970233876"} Oct 31 00:03:01.845812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:01.845801 4969 scope.go:122] "RemoveContainer" containerID="c232d385a18d29902ce4eb1872336791c06708d17fc48dd793f13235e41574cb" Oct 31 00:03:01.846037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:01.846018 4969 scope.go:122] "RemoveContainer" containerID="58528d26db0a15e451276149469bf9a6299eeebb5657f4c0921f095970233876" Oct 31 00:03:01.846237 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:03:01.846220 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:03:02.850727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:02.850696 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:03:02.851134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:02.851025 4969 scope.go:122] "RemoveContainer" containerID="58528d26db0a15e451276149469bf9a6299eeebb5657f4c0921f095970233876" Oct 31 00:03:02.851213 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:03:02.851195 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:03:04.612733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.612700 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:03:04.612733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.612737 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:03:04.613290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.612766 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:03:04.613290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.612814 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:03:04.613290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.612831 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:03:04.613290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.612859 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:03:04.615472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.615444 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8c82f1-cb80-4dae-a124-df184eab0f73-metrics-certs\") pod \"network-metrics-daemon-mgbdr\" (UID: \"af8c82f1-cb80-4dae-a124-df184eab0f73\") " pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:03:04.615821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.615795 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/32201f7b-9d56-4eb1-a3f4-b344b88beee9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9wfdj\" (UID: \"32201f7b-9d56-4eb1-a3f4-b344b88beee9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:03:04.615907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.615852 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb324747-4db5-4661-a18e-35933b8ad5ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzjx9\" (UID: \"eb324747-4db5-4661-a18e-35933b8ad5ad\") " pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:03:04.616087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.616066 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"image-registry-869c89c69d-stxfd\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:03:04.616156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.616136 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fabccc0d-dbcd-4185-b0dd-1ba72182cc94-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sn7fw\" (UID: \"fabccc0d-dbcd-4185-b0dd-1ba72182cc94\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:03:04.616193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.616136 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"image-registry-794c4fd7c9-8n2f4\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:03:04.642882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.642856 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" Oct 31 00:03:04.644869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.644847 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:03:04.645107 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.645089 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" Oct 31 00:03:04.651268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.651253 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgbdr" Oct 31 00:03:04.654173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.654154 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:03:04.656806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.656738 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzjx9" Oct 31 00:03:04.860045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.860017 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:03:04.860379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:04.860367 4969 scope.go:122] "RemoveContainer" containerID="58528d26db0a15e451276149469bf9a6299eeebb5657f4c0921f095970233876" Oct 31 00:03:04.860552 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:03:04.860534 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-wqpvv_openshift-console-operator(a00d1d31-58a5-4835-ad53-56c65f5adbb0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" Oct 31 00:03:05.036662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.036627 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw"] Oct 31 00:03:05.075753 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:03:05.075726 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabccc0d_dbcd_4185_b0dd_1ba72182cc94.slice/crio-0d0d11f5123ea32129269a84d5942a6926ae3db888a4b81774cfa16b98173d87 WatchSource:0}: Error finding container 0d0d11f5123ea32129269a84d5942a6926ae3db888a4b81774cfa16b98173d87: Status 404 returned error can't find the container with id 0d0d11f5123ea32129269a84d5942a6926ae3db888a4b81774cfa16b98173d87 Oct 31 00:03:05.086545 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.086519 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9wfdj"] Oct 31 00:03:05.100960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.100934 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869c89c69d-stxfd"] Oct 31 00:03:05.123694 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:03:05.123667 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32201f7b_9d56_4eb1_a3f4_b344b88beee9.slice/crio-66df351c8636fe1f1cb8ed246d9d0864da4e83d3c6d040fbdc35bcf9dfe9516e WatchSource:0}: Error finding container 66df351c8636fe1f1cb8ed246d9d0864da4e83d3c6d040fbdc35bcf9dfe9516e: Status 404 returned error can't find the container with id 66df351c8636fe1f1cb8ed246d9d0864da4e83d3c6d040fbdc35bcf9dfe9516e Oct 31 00:03:05.124347 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:03:05.124324 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46dedb6_cf0a_489e_9094_75e60189c87f.slice/crio-f5c303c77cc3528aa1845a7a9c46ae8de443fdab3d3fda8de31b4bb1ac9a014e WatchSource:0}: Error finding container f5c303c77cc3528aa1845a7a9c46ae8de443fdab3d3fda8de31b4bb1ac9a014e: Status 404 returned error can't find the container with id f5c303c77cc3528aa1845a7a9c46ae8de443fdab3d3fda8de31b4bb1ac9a014e Oct 31 00:03:05.132664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.132641 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgbdr"] Oct 31 00:03:05.134465 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:03:05.134438 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8c82f1_cb80_4dae_a124_df184eab0f73.slice/crio-f4b26fb4a5de04f7e04675b9bd6f669d77f45493de2e5dd1649c94be4edb312a WatchSource:0}: Error finding container f4b26fb4a5de04f7e04675b9bd6f669d77f45493de2e5dd1649c94be4edb312a: Status 404 returned error can't find the container with id f4b26fb4a5de04f7e04675b9bd6f669d77f45493de2e5dd1649c94be4edb312a Oct 31 00:03:05.139980 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:03:05.139953 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb324747_4db5_4661_a18e_35933b8ad5ad.slice/crio-bf43fef2b1eeeb6616e1b27a8736efb9693ed1d27a5a300a288af92a9e0aecc3 WatchSource:0}: Error finding container bf43fef2b1eeeb6616e1b27a8736efb9693ed1d27a5a300a288af92a9e0aecc3: Status 404 returned error can't find the container with id bf43fef2b1eeeb6616e1b27a8736efb9693ed1d27a5a300a288af92a9e0aecc3 Oct 31 00:03:05.143481 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:03:05.143456 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2f5b61_0433_4481_9c79_064d608b8300.slice/crio-7f54701f3f1bc4e9f47e95f6f512424685e2e1b09f8f5db7334322d44684aa9d WatchSource:0}: Error finding container 7f54701f3f1bc4e9f47e95f6f512424685e2e1b09f8f5db7334322d44684aa9d: Status 404 returned error can't find the container with id 7f54701f3f1bc4e9f47e95f6f512424685e2e1b09f8f5db7334322d44684aa9d Oct 31 00:03:05.151421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.151013 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzjx9"] Oct 31 00:03:05.151818 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.151800 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-794c4fd7c9-8n2f4"] Oct 31 00:03:05.863808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.863773 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzjx9" event={"ID":"eb324747-4db5-4661-a18e-35933b8ad5ad","Type":"ContainerStarted","Data":"8f5a970b020b7c38a4ba2db15680c7fa674e4020d99ab0403c143fd75688325f"} Oct 31 00:03:05.864210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.863836 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzjx9" event={"ID":"eb324747-4db5-4661-a18e-35933b8ad5ad","Type":"ContainerStarted","Data":"bf43fef2b1eeeb6616e1b27a8736efb9693ed1d27a5a300a288af92a9e0aecc3"} Oct 31 00:03:05.865374 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.865346 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" event={"ID":"0b2f5b61-0433-4481-9c79-064d608b8300","Type":"ContainerStarted","Data":"8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab"} Oct 31 00:03:05.865507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.865385 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" event={"ID":"0b2f5b61-0433-4481-9c79-064d608b8300","Type":"ContainerStarted","Data":"7f54701f3f1bc4e9f47e95f6f512424685e2e1b09f8f5db7334322d44684aa9d"} Oct 31 00:03:05.865507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.865446 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:03:05.867048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.867024 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" event={"ID":"d46dedb6-cf0a-489e-9094-75e60189c87f","Type":"ContainerStarted","Data":"fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144"} Oct 31 00:03:05.867150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.867062 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" event={"ID":"d46dedb6-cf0a-489e-9094-75e60189c87f","Type":"ContainerStarted","Data":"f5c303c77cc3528aa1845a7a9c46ae8de443fdab3d3fda8de31b4bb1ac9a014e"} Oct 31 00:03:05.867150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.867111 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:03:05.868239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.868213 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" event={"ID":"fabccc0d-dbcd-4185-b0dd-1ba72182cc94","Type":"ContainerStarted","Data":"0d0d11f5123ea32129269a84d5942a6926ae3db888a4b81774cfa16b98173d87"} Oct 31 00:03:05.869278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.869258 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" event={"ID":"32201f7b-9d56-4eb1-a3f4-b344b88beee9","Type":"ContainerStarted","Data":"66df351c8636fe1f1cb8ed246d9d0864da4e83d3c6d040fbdc35bcf9dfe9516e"} Oct 31 00:03:05.870455 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:05.870435 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgbdr" event={"ID":"af8c82f1-cb80-4dae-a124-df184eab0f73","Type":"ContainerStarted","Data":"f4b26fb4a5de04f7e04675b9bd6f669d77f45493de2e5dd1649c94be4edb312a"} Oct 31 00:03:06.880754 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:06.878586 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzjx9" event={"ID":"eb324747-4db5-4661-a18e-35933b8ad5ad","Type":"ContainerStarted","Data":"704c70e9e4494341c8e03c69d306883b344e950c9a6f95b214aa2994b69aad59"} Oct 31 00:03:07.784056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:07.783972 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-869c89c69d-stxfd"] Oct 31 00:03:07.884815 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:07.884770 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sn7fw" event={"ID":"fabccc0d-dbcd-4185-b0dd-1ba72182cc94","Type":"ContainerStarted","Data":"504cbe426b4d852f9728de5f88afd4420c896d072428f949fba7fccfbaec45b1"} Oct 31 00:03:07.886765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:07.886736 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9wfdj" event={"ID":"32201f7b-9d56-4eb1-a3f4-b344b88beee9","Type":"ContainerStarted","Data":"2b049173cf4cc86d1ffb3f0f82849e32a5d239f25eec5f48a0d7f9e1258349b4"} Oct 31 00:03:07.889343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:07.889315 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgbdr" event={"ID":"af8c82f1-cb80-4dae-a124-df184eab0f73","Type":"ContainerStarted","Data":"26d1210c33758798e55ee8a29bd1b9676fb44a3ade0db80e07124442feba7aad"} Oct 31 00:03:07.889482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:07.889365 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgbdr" event={"ID":"af8c82f1-cb80-4dae-a124-df184eab0f73","Type":"ContainerStarted","Data":"754564b67986c7b9bed25118ae968d36eabd406998468b2cab5900b4ea56acc2"} Oct 31 00:03:08.895246 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:08.895205 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzjx9" event={"ID":"eb324747-4db5-4661-a18e-35933b8ad5ad","Type":"ContainerStarted","Data":"7f87b68ab3f3c08dc9fd7351f05e2776f0eb22601c724d3e16c416ccd782028b"} Oct 31 00:03:09.842784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:09.842747 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2hhtz" Oct 31 00:03:14.696356 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:14.696325 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4b5st" Oct 31 00:03:17.447162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:17.447126 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qgstv" Oct 31 00:03:19.186406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.186366 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s6rcr"] Oct 31 00:03:19.236075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.236045 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.337828 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.337799 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sm2z2\"" Oct 31 00:03:19.337828 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.337801 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:03:19.338046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.337928 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:03:19.338165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.338148 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:03:19.338165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.338161 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:03:19.339198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.339181 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-tls\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.339248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.339204 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27dea15a-9318-4398-a940-aa43ab973e44-metrics-client-ca\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.339248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.339229 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.339323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.339301 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27dea15a-9318-4398-a940-aa43ab973e44-sys\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.339359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.339328 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-wtmp\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.339359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.339352 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27dea15a-9318-4398-a940-aa43ab973e44-root\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.339429 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.339371 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5n5\" (UniqueName: \"kubernetes.io/projected/27dea15a-9318-4398-a940-aa43ab973e44-kube-api-access-tq5n5\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.339477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.339462 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-textfile\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.339515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.339494 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440699 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-wtmp\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440744 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27dea15a-9318-4398-a940-aa43ab973e44-root\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440768 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5n5\" (UniqueName: \"kubernetes.io/projected/27dea15a-9318-4398-a940-aa43ab973e44-kube-api-access-tq5n5\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440801 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-textfile\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440830 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440847 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27dea15a-9318-4398-a940-aa43ab973e44-root\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440860 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-tls\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440880 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27dea15a-9318-4398-a940-aa43ab973e44-metrics-client-ca\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440907 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-wtmp\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440914 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.440999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.440976 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27dea15a-9318-4398-a940-aa43ab973e44-sys\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.441454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.441066 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27dea15a-9318-4398-a940-aa43ab973e44-sys\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.441454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.441204 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-textfile\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.441948 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.441631 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27dea15a-9318-4398-a940-aa43ab973e44-metrics-client-ca\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.442618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.442574 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-accelerators-collector-config\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.443713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.443689 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.443829 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.443716 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27dea15a-9318-4398-a940-aa43ab973e44-node-exporter-tls\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.501337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.501315 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5n5\" (UniqueName: \"kubernetes.io/projected/27dea15a-9318-4398-a940-aa43ab973e44-kube-api-access-tq5n5\") pod \"node-exporter-s6rcr\" (UID: \"27dea15a-9318-4398-a940-aa43ab973e44\") " pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.587759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.587734 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s6rcr" Oct 31 00:03:19.616366 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:03:19.616342 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27dea15a_9318_4398_a940_aa43ab973e44.slice/crio-271625dae72f025cc6b3a816cd928fdb7e454330536cd8de4e232ee58ff541aa WatchSource:0}: Error finding container 271625dae72f025cc6b3a816cd928fdb7e454330536cd8de4e232ee58ff541aa: Status 404 returned error can't find the container with id 271625dae72f025cc6b3a816cd928fdb7e454330536cd8de4e232ee58ff541aa Oct 31 00:03:19.937282 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:19.937242 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6rcr" event={"ID":"27dea15a-9318-4398-a940-aa43ab973e44","Type":"ContainerStarted","Data":"271625dae72f025cc6b3a816cd928fdb7e454330536cd8de4e232ee58ff541aa"} Oct 31 00:03:21.947516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:21.947441 4969 generic.go:356] "Generic (PLEG): container finished" podID="27dea15a-9318-4398-a940-aa43ab973e44" containerID="2946e0e3aa40f1083522a784fc377076eafdf8a6e08d18d38caaa1dd98f372a8" exitCode=0 Oct 31 00:03:21.947516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:21.947510 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6rcr" event={"ID":"27dea15a-9318-4398-a940-aa43ab973e44","Type":"ContainerDied","Data":"2946e0e3aa40f1083522a784fc377076eafdf8a6e08d18d38caaa1dd98f372a8"} Oct 31 00:03:22.953979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:22.953940 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6rcr" event={"ID":"27dea15a-9318-4398-a940-aa43ab973e44","Type":"ContainerStarted","Data":"fb0537f29d07e11f86fca52bd7962059638830817d33b6c689e69e27a015e82d"} Oct 31 00:03:22.954358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:22.953999 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s6rcr" event={"ID":"27dea15a-9318-4398-a940-aa43ab973e44","Type":"ContainerStarted","Data":"8c9294faabab5dda7ff1942f3733dbdb4c119bb274eced838c12b8e28e219c1a"} Oct 31 00:03:23.072444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:23.072393 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s6rcr" podStartSLOduration=1.993204837 podStartE2EDuration="4.072380032s" totalImagesPullingTime="2.001499521s" totalInitContainerRuntime="77.675674ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:23.071894186 +0000 UTC m=+35.547252609" watchObservedRunningTime="2026-10-31 00:03:23.072380032 +0000 UTC m=+35.547738461" Oct 31 00:03:24.726798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:24.726769 4969 scope.go:122] "RemoveContainer" containerID="58528d26db0a15e451276149469bf9a6299eeebb5657f4c0921f095970233876" Oct 31 00:03:24.962684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:24.962659 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:03:24.962859 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:24.962754 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" event={"ID":"a00d1d31-58a5-4835-ad53-56c65f5adbb0","Type":"ContainerStarted","Data":"67172a71d1565090b550d2be64eb12d99e182a0805d91713196278fb04b2b253"} Oct 31 00:03:24.963034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:24.963011 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:03:24.964064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:24.964032 4969 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-wqpvv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.21:8443/readyz\": dial tcp 10.133.0.21:8443: connect: connection refused" start-of-body= Oct 31 00:03:24.964194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:24.964074 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" podUID="a00d1d31-58a5-4835-ad53-56c65f5adbb0" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.21:8443/readyz\": dial tcp 10.133.0.21:8443: connect: connection refused" Oct 31 00:03:25.970480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:25.970453 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-wqpvv" Oct 31 00:03:26.882666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:26.882639 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:03:27.895024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:27.894994 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:03:30.461075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:30.461041 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-794c4fd7c9-8n2f4"] Oct 31 00:03:32.726522 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:03:32.726474 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:03:32.912862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:32.912789 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" podUID="d46dedb6-cf0a-489e-9094-75e60189c87f" containerName="registry" containerID="cri-o://fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144" gracePeriod=30 Oct 31 00:03:33.165919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.165895 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:03:33.261155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.261108 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted\") pod \"d46dedb6-cf0a-489e-9094-75e60189c87f\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " Oct 31 00:03:33.261333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.261172 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets\") pod \"d46dedb6-cf0a-489e-9094-75e60189c87f\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " Oct 31 00:03:33.261333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.261216 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates\") pod \"d46dedb6-cf0a-489e-9094-75e60189c87f\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " Oct 31 00:03:33.261333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.261249 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token\") pod \"d46dedb6-cf0a-489e-9094-75e60189c87f\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " Oct 31 00:03:33.261333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.261312 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") pod \"d46dedb6-cf0a-489e-9094-75e60189c87f\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " Oct 31 00:03:33.261552 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.261368 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8\") pod \"d46dedb6-cf0a-489e-9094-75e60189c87f\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " Oct 31 00:03:33.261552 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.261401 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration\") pod \"d46dedb6-cf0a-489e-9094-75e60189c87f\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " Oct 31 00:03:33.261552 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.261436 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca\") pod \"d46dedb6-cf0a-489e-9094-75e60189c87f\" (UID: \"d46dedb6-cf0a-489e-9094-75e60189c87f\") " Oct 31 00:03:33.261741 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.261699 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates" pod "d46dedb6-cf0a-489e-9094-75e60189c87f" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:33.262288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.262244 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca" pod "d46dedb6-cf0a-489e-9094-75e60189c87f" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:33.263915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.263858 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets" pod "d46dedb6-cf0a-489e-9094-75e60189c87f" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:33.264254 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.264224 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token" pod "d46dedb6-cf0a-489e-9094-75e60189c87f" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:33.264352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.264226 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8" pod "d46dedb6-cf0a-489e-9094-75e60189c87f" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f"). InnerVolumeSpecName "kube-api-access-qgft8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:33.264406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.264357 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls" pod "d46dedb6-cf0a-489e-9094-75e60189c87f" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:33.264446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.264415 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration" pod "d46dedb6-cf0a-489e-9094-75e60189c87f" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:33.268682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.268562 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted" pod "d46dedb6-cf0a-489e-9094-75e60189c87f" (UID: "d46dedb6-cf0a-489e-9094-75e60189c87f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:33.362213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.362138 4969 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d46dedb6-cf0a-489e-9094-75e60189c87f-ca-trust-extracted\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:33.362213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.362171 4969 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-installation-pull-secrets\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:33.362213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.362183 4969 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-certificates\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:33.362213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.362194 4969 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-bound-sa-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:33.362213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.362202 4969 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-registry-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:33.362213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.362210 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgft8\" (UniqueName: \"kubernetes.io/projected/d46dedb6-cf0a-489e-9094-75e60189c87f-kube-api-access-qgft8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:33.362488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.362224 4969 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d46dedb6-cf0a-489e-9094-75e60189c87f-image-registry-private-configuration\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:33.362488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:33.362238 4969 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d46dedb6-cf0a-489e-9094-75e60189c87f-trusted-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:34.000155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:34.000077 4969 generic.go:356] "Generic (PLEG): container finished" podID="d46dedb6-cf0a-489e-9094-75e60189c87f" containerID="fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144" exitCode=0 Oct 31 00:03:34.000155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:34.000127 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" Oct 31 00:03:34.000155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:34.000134 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" event={"ID":"d46dedb6-cf0a-489e-9094-75e60189c87f","Type":"ContainerDied","Data":"fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144"} Oct 31 00:03:34.000633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:34.000173 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869c89c69d-stxfd" event={"ID":"d46dedb6-cf0a-489e-9094-75e60189c87f","Type":"ContainerDied","Data":"f5c303c77cc3528aa1845a7a9c46ae8de443fdab3d3fda8de31b4bb1ac9a014e"} Oct 31 00:03:34.000633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:34.000192 4969 scope.go:122] "RemoveContainer" containerID="fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144" Oct 31 00:03:34.008550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:34.008529 4969 scope.go:122] "RemoveContainer" containerID="fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144" Oct 31 00:03:34.008876 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:03:34.008843 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144\": container with ID starting with fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144 not found: ID does not exist" containerID="fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144" Oct 31 00:03:34.008962 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:34.008887 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144"} err="failed to get container status \"fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144\": rpc error: code = NotFound desc = could not find container \"fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144\": container with ID starting with fb8b2ee237d8c2ad45dc1f82a953ae1e25e30f2ec1ddee44d4fe2822e9371144 not found: ID does not exist" Oct 31 00:03:34.215404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:34.215363 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-869c89c69d-stxfd"] Oct 31 00:03:34.264846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:34.264773 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-869c89c69d-stxfd"] Oct 31 00:03:35.732370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:35.732337 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46dedb6-cf0a-489e-9094-75e60189c87f" path="/var/lib/kubelet/pods/d46dedb6-cf0a-489e-9094-75e60189c87f/volumes" Oct 31 00:03:54.081418 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:54.081384 4969 generic.go:356] "Generic (PLEG): container finished" podID="af42676c-2cbc-47a8-a5a9-f38976f43064" containerID="bd9f71749ee877d1716dd933921096fdba641ce51bf8d624a16c921522879d7b" exitCode=0 Oct 31 00:03:54.081881 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:54.081461 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" event={"ID":"af42676c-2cbc-47a8-a5a9-f38976f43064","Type":"ContainerDied","Data":"bd9f71749ee877d1716dd933921096fdba641ce51bf8d624a16c921522879d7b"} Oct 31 00:03:54.081881 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:54.081802 4969 scope.go:122] "RemoveContainer" containerID="bd9f71749ee877d1716dd933921096fdba641ce51bf8d624a16c921522879d7b" Oct 31 00:03:55.087252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.087215 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2gwk4" event={"ID":"af42676c-2cbc-47a8-a5a9-f38976f43064","Type":"ContainerStarted","Data":"c3b40627131a7c2e98982a74fb00a84c5bdc89d779b330c16a0b3d19c2cb23f7"} Oct 31 00:03:55.481033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.480995 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" podUID="0b2f5b61-0433-4481-9c79-064d608b8300" containerName="registry" containerID="cri-o://8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab" gracePeriod=30 Oct 31 00:03:55.736242 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.736187 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:03:55.736449 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.736430 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") pod \"0b2f5b61-0433-4481-9c79-064d608b8300\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " Oct 31 00:03:55.736500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.736468 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted\") pod \"0b2f5b61-0433-4481-9c79-064d608b8300\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " Oct 31 00:03:55.736500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.736487 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates\") pod \"0b2f5b61-0433-4481-9c79-064d608b8300\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " Oct 31 00:03:55.736571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.736539 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2\") pod \"0b2f5b61-0433-4481-9c79-064d608b8300\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " Oct 31 00:03:55.736571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.736558 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca\") pod \"0b2f5b61-0433-4481-9c79-064d608b8300\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " Oct 31 00:03:55.736690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.736584 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token\") pod \"0b2f5b61-0433-4481-9c79-064d608b8300\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " Oct 31 00:03:55.736690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.736658 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration\") pod \"0b2f5b61-0433-4481-9c79-064d608b8300\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " Oct 31 00:03:55.736785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.736692 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets\") pod \"0b2f5b61-0433-4481-9c79-064d608b8300\" (UID: \"0b2f5b61-0433-4481-9c79-064d608b8300\") " Oct 31 00:03:55.737114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.737066 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates" pod "0b2f5b61-0433-4481-9c79-064d608b8300" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:55.737233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.737153 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca" pod "0b2f5b61-0433-4481-9c79-064d608b8300" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:03:55.739166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.739138 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2" pod "0b2f5b61-0433-4481-9c79-064d608b8300" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300"). InnerVolumeSpecName "kube-api-access-p5xd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:55.739377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.739355 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls" pod "0b2f5b61-0433-4481-9c79-064d608b8300" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:55.739650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.739622 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token" pod "0b2f5b61-0433-4481-9c79-064d608b8300" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:03:55.739756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.739741 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets" pod "0b2f5b61-0433-4481-9c79-064d608b8300" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:55.739827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.739807 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration" pod "0b2f5b61-0433-4481-9c79-064d608b8300" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:03:55.745744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.745683 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted" pod "0b2f5b61-0433-4481-9c79-064d608b8300" (UID: "0b2f5b61-0433-4481-9c79-064d608b8300"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:03:55.838126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.838088 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5xd2\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-kube-api-access-p5xd2\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:55.838126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.838118 4969 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-trusted-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:55.838126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.838128 4969 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-bound-sa-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:55.838377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.838139 4969 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-image-registry-private-configuration\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:55.838377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.838150 4969 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b2f5b61-0433-4481-9c79-064d608b8300-installation-pull-secrets\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:55.838377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.838158 4969 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b2f5b61-0433-4481-9c79-064d608b8300-registry-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:55.838377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.838166 4969 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b2f5b61-0433-4481-9c79-064d608b8300-ca-trust-extracted\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:55.838377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:55.838175 4969 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b2f5b61-0433-4481-9c79-064d608b8300-registry-certificates\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:03:56.092028 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:56.091946 4969 generic.go:356] "Generic (PLEG): container finished" podID="0b2f5b61-0433-4481-9c79-064d608b8300" containerID="8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab" exitCode=0 Oct 31 00:03:56.092028 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:56.092011 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" Oct 31 00:03:56.092522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:56.092012 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" event={"ID":"0b2f5b61-0433-4481-9c79-064d608b8300","Type":"ContainerDied","Data":"8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab"} Oct 31 00:03:56.092522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:56.092122 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794c4fd7c9-8n2f4" event={"ID":"0b2f5b61-0433-4481-9c79-064d608b8300","Type":"ContainerDied","Data":"7f54701f3f1bc4e9f47e95f6f512424685e2e1b09f8f5db7334322d44684aa9d"} Oct 31 00:03:56.092522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:56.092149 4969 scope.go:122] "RemoveContainer" containerID="8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab" Oct 31 00:03:56.101332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:56.101314 4969 scope.go:122] "RemoveContainer" containerID="8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab" Oct 31 00:03:56.101573 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:03:56.101555 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab\": container with ID starting with 8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab not found: ID does not exist" containerID="8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab" Oct 31 00:03:56.101629 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:56.101583 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab"} err="failed to get container status \"8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab\": rpc error: code = NotFound desc = could not find container \"8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab\": container with ID starting with 8ac36cc308fe2dda896b312c925e4de88e3b6f0cf7055107655dee86ed2b75ab not found: ID does not exist" Oct 31 00:03:56.255839 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:56.255806 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-794c4fd7c9-8n2f4"] Oct 31 00:03:56.261813 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:56.261791 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-794c4fd7c9-8n2f4"] Oct 31 00:03:57.731559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:57.731526 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b2f5b61-0433-4481-9c79-064d608b8300" path="/var/lib/kubelet/pods/0b2f5b61-0433-4481-9c79-064d608b8300/volumes" Oct 31 00:03:59.106077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:59.106044 4969 generic.go:356] "Generic (PLEG): container finished" podID="0dc5ea4f-6317-48ca-b075-5a292d72148c" containerID="cc855b40a56c90efa6fc19ea584ba7279ae54fef463b09c6f3cdf7fd49a31a7a" exitCode=0 Oct 31 00:03:59.106546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:59.106121 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" event={"ID":"0dc5ea4f-6317-48ca-b075-5a292d72148c","Type":"ContainerDied","Data":"cc855b40a56c90efa6fc19ea584ba7279ae54fef463b09c6f3cdf7fd49a31a7a"} Oct 31 00:03:59.106546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:03:59.106463 4969 scope.go:122] "RemoveContainer" containerID="cc855b40a56c90efa6fc19ea584ba7279ae54fef463b09c6f3cdf7fd49a31a7a" Oct 31 00:04:00.111759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:00.111721 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ckr6c" event={"ID":"0dc5ea4f-6317-48ca-b075-5a292d72148c","Type":"ContainerStarted","Data":"b8957e11052bbfcbffbc1d50a71ec6ef9a84e8e7b4aec96326cda4a4edfb08fa"} Oct 31 00:04:02.726545 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:04:02.726508 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:04:03.125973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:03.125939 4969 generic.go:356] "Generic (PLEG): container finished" podID="f61f7ab1-7e07-4f35-9ae8-9e1012e44557" containerID="7819c510f427dddc89e1804abfbb46654f78be5a9f672fc143a946953efdcda3" exitCode=0 Oct 31 00:04:03.126175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:03.126023 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7rj9b" event={"ID":"f61f7ab1-7e07-4f35-9ae8-9e1012e44557","Type":"ContainerDied","Data":"7819c510f427dddc89e1804abfbb46654f78be5a9f672fc143a946953efdcda3"} Oct 31 00:04:03.126411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:03.126393 4969 scope.go:122] "RemoveContainer" containerID="7819c510f427dddc89e1804abfbb46654f78be5a9f672fc143a946953efdcda3" Oct 31 00:04:04.132372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:04.132338 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7rj9b" event={"ID":"f61f7ab1-7e07-4f35-9ae8-9e1012e44557","Type":"ContainerStarted","Data":"478a5f4fe604c0c8092c11127e5bdc3670a2bd0f83500ea6cdd114faf2790ead"} Oct 31 00:04:11.160975 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:11.160932 4969 generic.go:356] "Generic (PLEG): container finished" podID="8cb93632-a513-4916-8377-3e2987a32f3d" containerID="a8ec0d9ea672557c67f922f9721efa80387db4fc598f093eea60fa0efb223520" exitCode=0 Oct 31 00:04:11.161396 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:11.161008 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-867tw" event={"ID":"8cb93632-a513-4916-8377-3e2987a32f3d","Type":"ContainerDied","Data":"a8ec0d9ea672557c67f922f9721efa80387db4fc598f093eea60fa0efb223520"} Oct 31 00:04:12.302652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:12.302628 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:04:12.371008 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:12.370978 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv\" (UniqueName: \"kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv\") pod \"8cb93632-a513-4916-8377-3e2987a32f3d\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " Oct 31 00:04:12.371161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:12.371066 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca\" (UniqueName: \"kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca\") pod \"8cb93632-a513-4916-8377-3e2987a32f3d\" (UID: \"8cb93632-a513-4916-8377-3e2987a32f3d\") " Oct 31 00:04:12.371431 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:12.371396 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca" pod "8cb93632-a513-4916-8377-3e2987a32f3d" (UID: "8cb93632-a513-4916-8377-3e2987a32f3d"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:04:12.373326 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:12.373303 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv" pod "8cb93632-a513-4916-8377-3e2987a32f3d" (UID: "8cb93632-a513-4916-8377-3e2987a32f3d"). InnerVolumeSpecName "kube-api-access-msxlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:12.471660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:12.471551 4969 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8cb93632-a513-4916-8377-3e2987a32f3d-serviceca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.471660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:12.471582 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msxlv\" (UniqueName: \"kubernetes.io/projected/8cb93632-a513-4916-8377-3e2987a32f3d-kube-api-access-msxlv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:04:13.169749 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:13.169717 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-867tw" Oct 31 00:04:13.169749 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:13.169735 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-867tw" event={"ID":"8cb93632-a513-4916-8377-3e2987a32f3d","Type":"ContainerDied","Data":"aaf64dda097e48686e67c23d3df1e4e9849995ae9434517fcef5adfc377bf919"} Oct 31 00:04:13.169953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:04:13.169780 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf64dda097e48686e67c23d3df1e4e9849995ae9434517fcef5adfc377bf919" Oct 31 00:04:32.726331 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:04:32.726290 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:05:02.726253 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:05:02.726167 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:05:32.725607 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:05:32.725547 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:06:02.726555 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:06:02.726515 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:06:32.724878 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:06:32.724842 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:07:02.725757 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:07:02.725722 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:07:32.725791 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:07:32.725754 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:07:47.736225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:07:47.736194 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:07:47.739759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:07:47.739737 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:08:02.726567 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:08:02.726488 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:08:32.725824 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:08:32.725788 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:09:02.725336 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:09:02.725298 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:09:15.974193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974164 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb"] Oct 31 00:09:15.974825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974569 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8cb93632-a513-4916-8377-3e2987a32f3d" containerName="image-pruner" Oct 31 00:09:15.974825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974585 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb93632-a513-4916-8377-3e2987a32f3d" containerName="image-pruner" Oct 31 00:09:15.974825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974624 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d46dedb6-cf0a-489e-9094-75e60189c87f" containerName="registry" Oct 31 00:09:15.974825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974632 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46dedb6-cf0a-489e-9094-75e60189c87f" containerName="registry" Oct 31 00:09:15.974825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974674 4969 cpu_manager.go:414] "containerMap: removing container" podUID="0b2f5b61-0433-4481-9c79-064d608b8300" containerName="registry" Oct 31 00:09:15.974825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974681 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2f5b61-0433-4481-9c79-064d608b8300" containerName="registry" Oct 31 00:09:15.974825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974772 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2f5b61-0433-4481-9c79-064d608b8300" containerName="registry" Oct 31 00:09:15.974825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974784 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb93632-a513-4916-8377-3e2987a32f3d" containerName="image-pruner" Oct 31 00:09:15.974825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.974793 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46dedb6-cf0a-489e-9094-75e60189c87f" containerName="registry" Oct 31 00:09:15.978109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:15.978088 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.039287 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.039254 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.039421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.039312 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hpf\" (UniqueName: \"kubernetes.io/projected/8d86f964-fe4e-417e-928e-56f38b0b84d8-kube-api-access-99hpf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.039421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.039384 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.045097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.045070 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:09:16.094855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.094825 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:09:16.095027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.095011 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-rzzq5\"" Oct 31 00:09:16.096230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.096212 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb"] Oct 31 00:09:16.140005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.139971 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hpf\" (UniqueName: \"kubernetes.io/projected/8d86f964-fe4e-417e-928e-56f38b0b84d8-kube-api-access-99hpf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.140145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.140015 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.140145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.140095 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.140476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.140453 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.140550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.140461 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.202227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.202197 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hpf\" (UniqueName: \"kubernetes.io/projected/8d86f964-fe4e-417e-928e-56f38b0b84d8-kube-api-access-99hpf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.293303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.293229 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:16.666885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.666860 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb"] Oct 31 00:09:16.669285 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:09:16.669258 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d86f964_fe4e_417e_928e_56f38b0b84d8.slice/crio-47f67d24beef689e616b7b88d1c504d1a36f2b64e2f43489467b8865bf73ce9b WatchSource:0}: Error finding container 47f67d24beef689e616b7b88d1c504d1a36f2b64e2f43489467b8865bf73ce9b: Status 404 returned error can't find the container with id 47f67d24beef689e616b7b88d1c504d1a36f2b64e2f43489467b8865bf73ce9b Oct 31 00:09:16.671287 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.671272 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:09:16.885443 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.885416 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2"] Oct 31 00:09:16.889951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.889935 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:16.945898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.945800 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-mr5rq\"" Oct 31 00:09:16.947195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.947174 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2"] Oct 31 00:09:16.950151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.950117 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlqwf\" (UniqueName: \"kubernetes.io/projected/4fe5853f-e254-4351-b216-9a2c89a1a306-kube-api-access-wlqwf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:16.950274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.950210 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:16.950274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:16.950243 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:17.051693 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.051655 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlqwf\" (UniqueName: \"kubernetes.io/projected/4fe5853f-e254-4351-b216-9a2c89a1a306-kube-api-access-wlqwf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:17.051693 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.051698 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:17.052114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.051715 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:17.052114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.052023 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:17.052114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.052046 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:17.123583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.123553 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlqwf\" (UniqueName: \"kubernetes.io/projected/4fe5853f-e254-4351-b216-9a2c89a1a306-kube-api-access-wlqwf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:17.205762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.205674 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:09:17.259340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.259298 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" event={"ID":"8d86f964-fe4e-417e-928e-56f38b0b84d8","Type":"ContainerStarted","Data":"47f67d24beef689e616b7b88d1c504d1a36f2b64e2f43489467b8865bf73ce9b"} Oct 31 00:09:17.600955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.600903 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2"] Oct 31 00:09:17.770006 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.769974 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg"] Oct 31 00:09:17.773384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.773368 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:17.828232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.828208 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-pkds7\"" Oct 31 00:09:17.851217 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.851151 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg"] Oct 31 00:09:17.858669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.858612 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnfl\" (UniqueName: \"kubernetes.io/projected/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-kube-api-access-jlnfl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:17.858811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.858723 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:17.858811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.858751 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:17.960093 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.960063 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:17.960093 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.960100 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:17.960347 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.960200 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnfl\" (UniqueName: \"kubernetes.io/projected/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-kube-api-access-jlnfl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:17.960543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.960521 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:17.960631 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:17.960574 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:18.004043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.003995 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5"] Oct 31 00:09:18.007773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.007749 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.024352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.024319 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnfl\" (UniqueName: \"kubernetes.io/projected/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-kube-api-access-jlnfl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:18.061520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.061485 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.061914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.061554 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2wv\" (UniqueName: \"kubernetes.io/projected/670c7343-c47d-40b1-9dd1-d7be7a3c7759-kube-api-access-wv2wv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.061914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.061660 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.063917 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.063894 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-k9ccw\"" Oct 31 00:09:18.072761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.072735 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5"] Oct 31 00:09:18.089860 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.089835 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:09:18.135384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.135356 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh"] Oct 31 00:09:18.139962 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.139935 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.165289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.164935 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2wv\" (UniqueName: \"kubernetes.io/projected/670c7343-c47d-40b1-9dd1-d7be7a3c7759-kube-api-access-wv2wv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.165289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.164987 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.165289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.165013 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.165289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.165034 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.165289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.165067 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.165289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.165084 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssqkr\" (UniqueName: \"kubernetes.io/projected/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-kube-api-access-ssqkr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.165925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.165901 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.166016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.165950 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.192412 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.192382 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kb7dw\"" Oct 31 00:09:18.205394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.205350 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh"] Oct 31 00:09:18.221926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.221899 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn"] Oct 31 00:09:18.226123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.226096 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.229102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.229080 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2wv\" (UniqueName: \"kubernetes.io/projected/670c7343-c47d-40b1-9dd1-d7be7a3c7759-kube-api-access-wv2wv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.264808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.264774 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" event={"ID":"4fe5853f-e254-4351-b216-9a2c89a1a306","Type":"ContainerStarted","Data":"2c887c0487759390cadce040632452689d81251b937aaa702196b679e0593047"} Oct 31 00:09:18.266114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.266092 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.266229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.266121 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssqkr\" (UniqueName: \"kubernetes.io/projected/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-kube-api-access-ssqkr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.266229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.266151 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6bqx\" (UniqueName: \"kubernetes.io/projected/5571790b-04b2-49d1-93b7-8918952ace4b-kube-api-access-r6bqx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.266229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.266203 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.266406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.266238 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.266406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.266358 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.266549 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.266522 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.266729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.266710 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.278776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.278743 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-mwzmv\"" Oct 31 00:09:18.287017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.286992 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn"] Oct 31 00:09:18.325709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.325676 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:09:18.340557 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.340472 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx"] Oct 31 00:09:18.342649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.342625 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssqkr\" (UniqueName: \"kubernetes.io/projected/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-kube-api-access-ssqkr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.344648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.344626 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.367309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.367273 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqvjx\" (UniqueName: \"kubernetes.io/projected/401a3887-09c5-4192-a7df-4c758b19ac8e-kube-api-access-mqvjx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.367420 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.367333 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6bqx\" (UniqueName: \"kubernetes.io/projected/5571790b-04b2-49d1-93b7-8918952ace4b-kube-api-access-r6bqx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.367492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.367416 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.367492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.367463 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.367749 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.367500 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.367749 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.367522 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.367865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.367848 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.367957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.367939 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.397621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.397574 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2x5zb\"" Oct 31 00:09:18.407511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.407483 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx"] Oct 31 00:09:18.434610 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.434571 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6bqx\" (UniqueName: \"kubernetes.io/projected/5571790b-04b2-49d1-93b7-8918952ace4b-kube-api-access-r6bqx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.468959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.468924 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.469114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.468968 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.469114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.469012 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqvjx\" (UniqueName: \"kubernetes.io/projected/401a3887-09c5-4192-a7df-4c758b19ac8e-kube-api-access-mqvjx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.469485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.469462 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.469617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.469578 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.473741 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.473722 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:09:18.507577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.507543 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg"] Oct 31 00:09:18.511566 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:09:18.511531 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc427ff7_e582_4b9d_92bb_8ca28e76fbfa.slice/crio-bbae5128ab5aab8bf5aa25d9240a597445522b8f785934bffc81df167fed5122 WatchSource:0}: Error finding container bbae5128ab5aab8bf5aa25d9240a597445522b8f785934bffc81df167fed5122: Status 404 returned error can't find the container with id bbae5128ab5aab8bf5aa25d9240a597445522b8f785934bffc81df167fed5122 Oct 31 00:09:18.533326 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.533292 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqvjx\" (UniqueName: \"kubernetes.io/projected/401a3887-09c5-4192-a7df-4c758b19ac8e-kube-api-access-mqvjx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.553198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.553025 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:09:18.668209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.668135 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:09:18.719127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.719055 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5"] Oct 31 00:09:18.853341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.853316 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh"] Oct 31 00:09:18.856465 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:09:18.856430 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a71577e_7c5f_4311_a5c7_3ebe07425f6b.slice/crio-b414a831afcad3bda0c11c1a12ca0959698e0bf3752e170dd30ac99df5d37736 WatchSource:0}: Error finding container b414a831afcad3bda0c11c1a12ca0959698e0bf3752e170dd30ac99df5d37736: Status 404 returned error can't find the container with id b414a831afcad3bda0c11c1a12ca0959698e0bf3752e170dd30ac99df5d37736 Oct 31 00:09:18.941384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:18.941303 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn"] Oct 31 00:09:18.949711 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:09:18.949666 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5571790b_04b2_49d1_93b7_8918952ace4b.slice/crio-6faaca36bd740535283534e19c424e4810a08a2d45f3451b8e9a2f75379a1949 WatchSource:0}: Error finding container 6faaca36bd740535283534e19c424e4810a08a2d45f3451b8e9a2f75379a1949: Status 404 returned error can't find the container with id 6faaca36bd740535283534e19c424e4810a08a2d45f3451b8e9a2f75379a1949 Oct 31 00:09:19.053363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:19.053333 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx"] Oct 31 00:09:19.055428 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:09:19.055402 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401a3887_09c5_4192_a7df_4c758b19ac8e.slice/crio-f93d2188e2dc33188b098c959fceefcf266ac5514bfff2ce9639101acee2e4f7 WatchSource:0}: Error finding container f93d2188e2dc33188b098c959fceefcf266ac5514bfff2ce9639101acee2e4f7: Status 404 returned error can't find the container with id f93d2188e2dc33188b098c959fceefcf266ac5514bfff2ce9639101acee2e4f7 Oct 31 00:09:19.272445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:19.272328 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" event={"ID":"5571790b-04b2-49d1-93b7-8918952ace4b","Type":"ContainerStarted","Data":"6faaca36bd740535283534e19c424e4810a08a2d45f3451b8e9a2f75379a1949"} Oct 31 00:09:19.274002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:19.273944 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" event={"ID":"401a3887-09c5-4192-a7df-4c758b19ac8e","Type":"ContainerStarted","Data":"f93d2188e2dc33188b098c959fceefcf266ac5514bfff2ce9639101acee2e4f7"} Oct 31 00:09:19.275209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:19.275181 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" event={"ID":"670c7343-c47d-40b1-9dd1-d7be7a3c7759","Type":"ContainerStarted","Data":"3f694f73707eedbd4bfae0a790fd5b211ee4d3b11a57f0527ac4684fb77837c0"} Oct 31 00:09:19.276918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:19.276882 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" event={"ID":"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa","Type":"ContainerStarted","Data":"bbae5128ab5aab8bf5aa25d9240a597445522b8f785934bffc81df167fed5122"} Oct 31 00:09:19.278805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:19.278783 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" event={"ID":"2a71577e-7c5f-4311-a5c7-3ebe07425f6b","Type":"ContainerStarted","Data":"b414a831afcad3bda0c11c1a12ca0959698e0bf3752e170dd30ac99df5d37736"} Oct 31 00:09:21.428084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.428047 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k"] Oct 31 00:09:21.433201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.432286 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.495761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.495666 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.495761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.495752 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.496014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.495789 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kk5w\" (UniqueName: \"kubernetes.io/projected/901df951-0dd8-4fad-8291-54d04507d67b-kube-api-access-6kk5w\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.537316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.537282 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k"] Oct 31 00:09:21.596726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.596686 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.596889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.596748 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.596889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.596782 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kk5w\" (UniqueName: \"kubernetes.io/projected/901df951-0dd8-4fad-8291-54d04507d67b-kube-api-access-6kk5w\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.597751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.597725 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.597976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.597923 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.663750 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.663716 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kk5w\" (UniqueName: \"kubernetes.io/projected/901df951-0dd8-4fad-8291-54d04507d67b-kube-api-access-6kk5w\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:21.755104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:21.755028 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:09:22.230215 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:22.230182 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k"] Oct 31 00:09:22.237241 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:09:22.237213 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901df951_0dd8_4fad_8291_54d04507d67b.slice/crio-823b887387e189a7090cbf49c492d50de942261df6fe4df1da7f35dd16439b17 WatchSource:0}: Error finding container 823b887387e189a7090cbf49c492d50de942261df6fe4df1da7f35dd16439b17: Status 404 returned error can't find the container with id 823b887387e189a7090cbf49c492d50de942261df6fe4df1da7f35dd16439b17 Oct 31 00:09:22.312797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:22.312761 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" event={"ID":"901df951-0dd8-4fad-8291-54d04507d67b","Type":"ContainerStarted","Data":"823b887387e189a7090cbf49c492d50de942261df6fe4df1da7f35dd16439b17"} Oct 31 00:09:23.458601 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.458550 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr"] Oct 31 00:09:23.462508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.462486 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.515077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.515041 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.515233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.515096 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcqdf\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-kube-api-access-rcqdf\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.515233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.515144 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.515233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.515175 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.515233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.515205 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.515461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.515234 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.515461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.515308 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.515461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.515364 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.515461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.515388 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6e4058-e548-4a56-9017-5b57dad353ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.519101 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.519077 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 00:09:23.535466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.535380 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-r4dfh\"" Oct 31 00:09:23.572928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.572888 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr"] Oct 31 00:09:23.616957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.616920 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617132 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.616969 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6e4058-e548-4a56-9017-5b57dad353ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617132 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617088 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617129 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcqdf\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-kube-api-access-rcqdf\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617181 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617213 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617388 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617264 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617388 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617293 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617388 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617375 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617378 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617445 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617732 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.617952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617927 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6e4058-e548-4a56-9017-5b57dad353ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.618026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.617977 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.636208 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.636119 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.638779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.638734 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.701176 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.701112 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.705293 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.705262 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcqdf\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-kube-api-access-rcqdf\") pod \"maas-default-gateway-openshift-default-58c6585447-thkzr\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:23.783007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:23.782895 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:24.281342 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:24.281305 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr"] Oct 31 00:09:24.338665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:24.338628 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" event={"ID":"2f6e4058-e548-4a56-9017-5b57dad353ce","Type":"ContainerStarted","Data":"79460c2d57d922453b68791cf10e6942f4ef76e77bcd88c58ad199b9d5aa050b"} Oct 31 00:09:25.613210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.613052 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd"] Oct 31 00:09:25.619497 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.619476 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.647425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.645798 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz"] Oct 31 00:09:25.650828 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.650440 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.678054 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.677897 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd"] Oct 31 00:09:25.736462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.736433 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz"] Oct 31 00:09:25.738991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.738965 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.739105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.739004 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.739105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.739069 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqpg\" (UniqueName: \"kubernetes.io/projected/8eb27f0e-86c5-4611-a807-eb05397eefd0-kube-api-access-mdqpg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.739105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.739099 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.739272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.739148 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.739272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.739189 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7dl\" (UniqueName: \"kubernetes.io/projected/fc66fc26-a5d3-4165-8937-b2dae564dab9-kube-api-access-4x7dl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.840086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.840052 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqpg\" (UniqueName: \"kubernetes.io/projected/8eb27f0e-86c5-4611-a807-eb05397eefd0-kube-api-access-mdqpg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.840264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.840095 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.840264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.840132 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.840264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.840164 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7dl\" (UniqueName: \"kubernetes.io/projected/fc66fc26-a5d3-4165-8937-b2dae564dab9-kube-api-access-4x7dl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.840264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.840202 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.840264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.840216 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.840541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.840512 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.841062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.841038 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.841240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.841224 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.841642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.841619 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.912043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.912008 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqpg\" (UniqueName: \"kubernetes.io/projected/8eb27f0e-86c5-4611-a807-eb05397eefd0-kube-api-access-mdqpg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.921871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.921813 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7dl\" (UniqueName: \"kubernetes.io/projected/fc66fc26-a5d3-4165-8937-b2dae564dab9-kube-api-access-4x7dl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:25.941685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.941657 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:09:25.972584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:25.972387 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:09:26.404653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:26.404616 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd"] Oct 31 00:09:26.413730 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:09:26.413689 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb27f0e_86c5_4611_a807_eb05397eefd0.slice/crio-24c4f06d6f61e9f634acf48d723aac3b423553195df24aa7d93b91af308f88ee WatchSource:0}: Error finding container 24c4f06d6f61e9f634acf48d723aac3b423553195df24aa7d93b91af308f88ee: Status 404 returned error can't find the container with id 24c4f06d6f61e9f634acf48d723aac3b423553195df24aa7d93b91af308f88ee Oct 31 00:09:26.428687 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:26.427674 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz"] Oct 31 00:09:26.432772 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:09:26.432738 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc66fc26_a5d3_4165_8937_b2dae564dab9.slice/crio-aec88f72198ec258d3c1a21ee498d37a7c6f450dc5f904e3cd401d3d70ce0344 WatchSource:0}: Error finding container aec88f72198ec258d3c1a21ee498d37a7c6f450dc5f904e3cd401d3d70ce0344: Status 404 returned error can't find the container with id aec88f72198ec258d3c1a21ee498d37a7c6f450dc5f904e3cd401d3d70ce0344 Oct 31 00:09:27.401855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:27.401766 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" event={"ID":"fc66fc26-a5d3-4165-8937-b2dae564dab9","Type":"ContainerStarted","Data":"aec88f72198ec258d3c1a21ee498d37a7c6f450dc5f904e3cd401d3d70ce0344"} Oct 31 00:09:27.404174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:27.404101 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" event={"ID":"8eb27f0e-86c5-4611-a807-eb05397eefd0","Type":"ContainerStarted","Data":"24c4f06d6f61e9f634acf48d723aac3b423553195df24aa7d93b91af308f88ee"} Oct 31 00:09:32.781199 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:09:32.781120 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:09:33.488030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:33.487988 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" event={"ID":"8d86f964-fe4e-417e-928e-56f38b0b84d8","Type":"ContainerStarted","Data":"c0bf31e3f05e82721351f2c6dc2595b5dedc1fb4967a755f9a33667b1373b32c"} Oct 31 00:09:34.506612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:34.506565 4969 generic.go:356] "Generic (PLEG): container finished" podID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerID="c0bf31e3f05e82721351f2c6dc2595b5dedc1fb4967a755f9a33667b1373b32c" exitCode=0 Oct 31 00:09:34.507085 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:34.506634 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" event={"ID":"8d86f964-fe4e-417e-928e-56f38b0b84d8","Type":"ContainerDied","Data":"c0bf31e3f05e82721351f2c6dc2595b5dedc1fb4967a755f9a33667b1373b32c"} Oct 31 00:09:35.518579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:35.518227 4969 generic.go:356] "Generic (PLEG): container finished" podID="901df951-0dd8-4fad-8291-54d04507d67b" containerID="8fbb90bf33adf19523afb9e8bbbc2824e77f5ddc6aeef9fa8f6328c153f23161" exitCode=0 Oct 31 00:09:35.518579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:35.518285 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" event={"ID":"901df951-0dd8-4fad-8291-54d04507d67b","Type":"ContainerDied","Data":"8fbb90bf33adf19523afb9e8bbbc2824e77f5ddc6aeef9fa8f6328c153f23161"} Oct 31 00:09:35.907481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:35.907430 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:09:35.907481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:35.907519 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:09:35.907754 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:35.907563 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:09:36.527977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:36.527939 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" event={"ID":"2f6e4058-e548-4a56-9017-5b57dad353ce","Type":"ContainerStarted","Data":"f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc"} Oct 31 00:09:36.532516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:36.531779 4969 generic.go:356] "Generic (PLEG): container finished" podID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerID="8da8b3063b0bce8b7f18fc4a8b7bc90060fd9e5e75276fff8644fed18e14fc51" exitCode=0 Oct 31 00:09:36.532516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:36.531905 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" event={"ID":"4fe5853f-e254-4351-b216-9a2c89a1a306","Type":"ContainerDied","Data":"8da8b3063b0bce8b7f18fc4a8b7bc90060fd9e5e75276fff8644fed18e14fc51"} Oct 31 00:09:36.653780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:36.653227 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" podStartSLOduration=2.037745749 podStartE2EDuration="13.65320985s" totalImagesPullingTime="11.615464101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:36.651040056 +0000 UTC m=+409.126398479" watchObservedRunningTime="2026-10-31 00:09:36.65320985 +0000 UTC m=+409.128568274" Oct 31 00:09:36.783912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:36.783867 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:37.541261 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:37.541228 4969 generic.go:356] "Generic (PLEG): container finished" podID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerID="2278a857477b7bb613dbf5c14882993b0a344268cdbdb3f27cbe74c68ca7eef0" exitCode=0 Oct 31 00:09:37.541795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:37.541330 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" event={"ID":"670c7343-c47d-40b1-9dd1-d7be7a3c7759","Type":"ContainerDied","Data":"2278a857477b7bb613dbf5c14882993b0a344268cdbdb3f27cbe74c68ca7eef0"} Oct 31 00:09:37.546019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:37.545986 4969 generic.go:356] "Generic (PLEG): container finished" podID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerID="66b33cd57cede7abe2ad25d74af87b73f16cecfa37b4fb7a42b860c9749119a4" exitCode=0 Oct 31 00:09:37.546705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:37.546685 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" event={"ID":"8d86f964-fe4e-417e-928e-56f38b0b84d8","Type":"ContainerDied","Data":"66b33cd57cede7abe2ad25d74af87b73f16cecfa37b4fb7a42b860c9749119a4"} Oct 31 00:09:37.790584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:37.790554 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:38.552490 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.552455 4969 generic.go:356] "Generic (PLEG): container finished" podID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerID="62d75197293ab519d211ca4358be64a5c7be6a24d10a31d875adeaa87bc2bbfe" exitCode=0 Oct 31 00:09:38.552955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.552567 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" event={"ID":"8eb27f0e-86c5-4611-a807-eb05397eefd0","Type":"ContainerDied","Data":"62d75197293ab519d211ca4358be64a5c7be6a24d10a31d875adeaa87bc2bbfe"} Oct 31 00:09:38.554908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.554882 4969 generic.go:356] "Generic (PLEG): container finished" podID="5571790b-04b2-49d1-93b7-8918952ace4b" containerID="3edc411a470f9a848c0c45e7a7d7ef058e87a70071ab436ca0458ccbc1ff59c9" exitCode=0 Oct 31 00:09:38.554996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.554966 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" event={"ID":"5571790b-04b2-49d1-93b7-8918952ace4b","Type":"ContainerDied","Data":"3edc411a470f9a848c0c45e7a7d7ef058e87a70071ab436ca0458ccbc1ff59c9"} Oct 31 00:09:38.556957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.556932 4969 generic.go:356] "Generic (PLEG): container finished" podID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerID="3c7934ac414ae446e697db5e3aee3681b049f69b95ada20f12dd5063184efe8e" exitCode=0 Oct 31 00:09:38.557040 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.556966 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" event={"ID":"401a3887-09c5-4192-a7df-4c758b19ac8e","Type":"ContainerDied","Data":"3c7934ac414ae446e697db5e3aee3681b049f69b95ada20f12dd5063184efe8e"} Oct 31 00:09:38.558979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.558958 4969 generic.go:356] "Generic (PLEG): container finished" podID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerID="6f4bc5c69002708b84896492d6ae97d34052910f7a5c6e58088d36291ca068ec" exitCode=0 Oct 31 00:09:38.559065 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.559010 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" event={"ID":"fc66fc26-a5d3-4165-8937-b2dae564dab9","Type":"ContainerDied","Data":"6f4bc5c69002708b84896492d6ae97d34052910f7a5c6e58088d36291ca068ec"} Oct 31 00:09:38.560759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.560656 4969 generic.go:356] "Generic (PLEG): container finished" podID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerID="8c97279d5fce42d561e18363a44290159616d3ba6c1b65a219d38eb24fbdd901" exitCode=0 Oct 31 00:09:38.560759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.560728 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" event={"ID":"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa","Type":"ContainerDied","Data":"8c97279d5fce42d561e18363a44290159616d3ba6c1b65a219d38eb24fbdd901"} Oct 31 00:09:38.562951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.562815 4969 generic.go:356] "Generic (PLEG): container finished" podID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerID="91299c3e5d513c1bbe57b911d4566da47dc63b214c00cfd7dc590302e60eed37" exitCode=0 Oct 31 00:09:38.562951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.562899 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" event={"ID":"2a71577e-7c5f-4311-a5c7-3ebe07425f6b","Type":"ContainerDied","Data":"91299c3e5d513c1bbe57b911d4566da47dc63b214c00cfd7dc590302e60eed37"} Oct 31 00:09:38.565785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.565762 4969 generic.go:356] "Generic (PLEG): container finished" podID="901df951-0dd8-4fad-8291-54d04507d67b" containerID="c97b5b6041a8ad01da3b29ec36e793afa15fc248c48edd83564819b0f552cb72" exitCode=0 Oct 31 00:09:38.565888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.565869 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" event={"ID":"901df951-0dd8-4fad-8291-54d04507d67b","Type":"ContainerDied","Data":"c97b5b6041a8ad01da3b29ec36e793afa15fc248c48edd83564819b0f552cb72"} Oct 31 00:09:38.566090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.566056 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:38.567439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:38.567422 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:09:40.584119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.584030 4969 generic.go:356] "Generic (PLEG): container finished" podID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerID="b0e3aad8b5f57ad902fceaf4b4add69be92647c7aea0c20d115194f727ea7553" exitCode=0 Oct 31 00:09:40.584575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.584153 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" event={"ID":"670c7343-c47d-40b1-9dd1-d7be7a3c7759","Type":"ContainerDied","Data":"b0e3aad8b5f57ad902fceaf4b4add69be92647c7aea0c20d115194f727ea7553"} Oct 31 00:09:40.593489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.593460 4969 generic.go:356] "Generic (PLEG): container finished" podID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerID="1e8c0dacddd4b485b2bc2ed6434cc540e31ac00c559ad63a517b5d6fce377bb3" exitCode=0 Oct 31 00:09:40.593617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.593546 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" event={"ID":"2a71577e-7c5f-4311-a5c7-3ebe07425f6b","Type":"ContainerDied","Data":"1e8c0dacddd4b485b2bc2ed6434cc540e31ac00c559ad63a517b5d6fce377bb3"} Oct 31 00:09:40.597560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.597330 4969 generic.go:356] "Generic (PLEG): container finished" podID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerID="1d3e5f6aeaf28324b5042081b77309ac5d89db3d08c66296999d52b87b614564" exitCode=0 Oct 31 00:09:40.597560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.597504 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" event={"ID":"8eb27f0e-86c5-4611-a807-eb05397eefd0","Type":"ContainerDied","Data":"1d3e5f6aeaf28324b5042081b77309ac5d89db3d08c66296999d52b87b614564"} Oct 31 00:09:40.603668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.603088 4969 generic.go:356] "Generic (PLEG): container finished" podID="5571790b-04b2-49d1-93b7-8918952ace4b" containerID="33df8e8c979feff3cfd986a859728eae730ede2160d865315a7177d0192cd829" exitCode=0 Oct 31 00:09:40.603668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.603184 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" event={"ID":"5571790b-04b2-49d1-93b7-8918952ace4b","Type":"ContainerDied","Data":"33df8e8c979feff3cfd986a859728eae730ede2160d865315a7177d0192cd829"} Oct 31 00:09:40.607067 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.607043 4969 generic.go:356] "Generic (PLEG): container finished" podID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerID="807437679e193406eeb31d7047377de9dec82fbaf5e60517a794cca4b95f3002" exitCode=0 Oct 31 00:09:40.607159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.607118 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" event={"ID":"401a3887-09c5-4192-a7df-4c758b19ac8e","Type":"ContainerDied","Data":"807437679e193406eeb31d7047377de9dec82fbaf5e60517a794cca4b95f3002"} Oct 31 00:09:40.611300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.611276 4969 generic.go:356] "Generic (PLEG): container finished" podID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerID="aad8e7c3644ce612653d226db1bee08da2e467f89cc1f22ce481e9a924e38063" exitCode=0 Oct 31 00:09:40.611376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:40.611356 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" event={"ID":"fc66fc26-a5d3-4165-8937-b2dae564dab9","Type":"ContainerDied","Data":"aad8e7c3644ce612653d226db1bee08da2e467f89cc1f22ce481e9a924e38063"} Oct 31 00:09:41.627535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:41.626550 4969 generic.go:356] "Generic (PLEG): container finished" podID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerID="ce932f29d0e65486e58060dadca945c0c3d175c5f2fb501c8c782b38ccb6cd87" exitCode=0 Oct 31 00:09:41.627535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:41.626672 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" event={"ID":"4fe5853f-e254-4351-b216-9a2c89a1a306","Type":"ContainerDied","Data":"ce932f29d0e65486e58060dadca945c0c3d175c5f2fb501c8c782b38ccb6cd87"} Oct 31 00:09:45.691733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:45.690445 4969 generic.go:356] "Generic (PLEG): container finished" podID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerID="f27b6c9796abb63f8f0dc8ee0b96fab5c0296368c12e7747a0213824c825a5fe" exitCode=0 Oct 31 00:09:45.691733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:45.690620 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" event={"ID":"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa","Type":"ContainerDied","Data":"f27b6c9796abb63f8f0dc8ee0b96fab5c0296368c12e7747a0213824c825a5fe"} Oct 31 00:09:55.858184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:55.858141 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" event={"ID":"8d86f964-fe4e-417e-928e-56f38b0b84d8","Type":"ContainerStarted","Data":"80828966d6ece8dc9fce3ae3ae7e72a74c632c0b1d10c5f3d9a4528c5ed1a874"} Oct 31 00:09:55.977939 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:55.977878 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" podStartSLOduration=4.460416907 podStartE2EDuration="40.977862753s" totalImagesPullingTime="36.517445846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:55.977563203 +0000 UTC m=+428.452921627" watchObservedRunningTime="2026-10-31 00:09:55.977862753 +0000 UTC m=+428.453221179" Oct 31 00:09:56.880785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:56.880749 4969 generic.go:356] "Generic (PLEG): container finished" podID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerID="80828966d6ece8dc9fce3ae3ae7e72a74c632c0b1d10c5f3d9a4528c5ed1a874" exitCode=0 Oct 31 00:09:56.881207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:56.880875 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" event={"ID":"8d86f964-fe4e-417e-928e-56f38b0b84d8","Type":"ContainerDied","Data":"80828966d6ece8dc9fce3ae3ae7e72a74c632c0b1d10c5f3d9a4528c5ed1a874"} Oct 31 00:09:58.113148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.112991 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:58.267055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.267018 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d86f964-fe4e-417e-928e-56f38b0b84d8-kube-api-access-99hpf\" (UniqueName: \"kubernetes.io/projected/8d86f964-fe4e-417e-928e-56f38b0b84d8-kube-api-access-99hpf\") pod \"8d86f964-fe4e-417e-928e-56f38b0b84d8\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " Oct 31 00:09:58.267234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.267084 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-bundle\") pod \"8d86f964-fe4e-417e-928e-56f38b0b84d8\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " Oct 31 00:09:58.267234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.267190 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-util\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-util\") pod \"8d86f964-fe4e-417e-928e-56f38b0b84d8\" (UID: \"8d86f964-fe4e-417e-928e-56f38b0b84d8\") " Oct 31 00:09:58.268773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.268739 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-bundle" pod "8d86f964-fe4e-417e-928e-56f38b0b84d8" (UID: "8d86f964-fe4e-417e-928e-56f38b0b84d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:58.272811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.272784 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d86f964-fe4e-417e-928e-56f38b0b84d8-kube-api-access-99hpf" pod "8d86f964-fe4e-417e-928e-56f38b0b84d8" (UID: "8d86f964-fe4e-417e-928e-56f38b0b84d8"). InnerVolumeSpecName "kube-api-access-99hpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:58.277051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.277019 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-util" pod "8d86f964-fe4e-417e-928e-56f38b0b84d8" (UID: "8d86f964-fe4e-417e-928e-56f38b0b84d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:58.369043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.368980 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99hpf\" (UniqueName: \"kubernetes.io/projected/8d86f964-fe4e-417e-928e-56f38b0b84d8-kube-api-access-99hpf\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:09:58.369043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.369017 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:09:58.369043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.369032 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d86f964-fe4e-417e-928e-56f38b0b84d8-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:09:58.922443 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.921365 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" event={"ID":"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa","Type":"ContainerStarted","Data":"32534734bf248c93ffd4f3295a2d73937eb3c557cd032bdcdbd6c1ce737cb216"} Oct 31 00:09:58.927654 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.925916 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" event={"ID":"8d86f964-fe4e-417e-928e-56f38b0b84d8","Type":"ContainerDied","Data":"47f67d24beef689e616b7b88d1c504d1a36f2b64e2f43489467b8865bf73ce9b"} Oct 31 00:09:58.927654 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.925971 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47f67d24beef689e616b7b88d1c504d1a36f2b64e2f43489467b8865bf73ce9b" Oct 31 00:09:58.927654 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.926057 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rkfvb" Oct 31 00:09:58.941981 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.941952 4969 generic.go:356] "Generic (PLEG): container finished" podID="901df951-0dd8-4fad-8291-54d04507d67b" containerID="a791417bb56aa144843ea7e2f658498da247a005ef705a653081185592f0ff88" exitCode=0 Oct 31 00:09:58.942106 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:58.942055 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" event={"ID":"901df951-0dd8-4fad-8291-54d04507d67b","Type":"ContainerDied","Data":"a791417bb56aa144843ea7e2f658498da247a005ef705a653081185592f0ff88"} Oct 31 00:09:59.042036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:59.040550 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" podStartSLOduration=4.381277696 podStartE2EDuration="42.040532501s" totalImagesPullingTime="37.659254805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:59.039910745 +0000 UTC m=+431.515269190" watchObservedRunningTime="2026-10-31 00:09:59.040532501 +0000 UTC m=+431.515890926" Oct 31 00:09:59.955710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:59.955670 4969 generic.go:356] "Generic (PLEG): container finished" podID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerID="32534734bf248c93ffd4f3295a2d73937eb3c557cd032bdcdbd6c1ce737cb216" exitCode=0 Oct 31 00:09:59.956225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:09:59.955795 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" event={"ID":"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa","Type":"ContainerDied","Data":"32534734bf248c93ffd4f3295a2d73937eb3c557cd032bdcdbd6c1ce737cb216"} Oct 31 00:10:00.162572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.162242 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:10:00.302975 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.302899 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-util\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-util\") pod \"901df951-0dd8-4fad-8291-54d04507d67b\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " Oct 31 00:10:00.302975 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.302951 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-bundle\") pod \"901df951-0dd8-4fad-8291-54d04507d67b\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " Oct 31 00:10:00.303210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.302992 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901df951-0dd8-4fad-8291-54d04507d67b-kube-api-access-6kk5w\" (UniqueName: \"kubernetes.io/projected/901df951-0dd8-4fad-8291-54d04507d67b-kube-api-access-6kk5w\") pod \"901df951-0dd8-4fad-8291-54d04507d67b\" (UID: \"901df951-0dd8-4fad-8291-54d04507d67b\") " Oct 31 00:10:00.303731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.303683 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-bundle" pod "901df951-0dd8-4fad-8291-54d04507d67b" (UID: "901df951-0dd8-4fad-8291-54d04507d67b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:00.310137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.310106 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-util" pod "901df951-0dd8-4fad-8291-54d04507d67b" (UID: "901df951-0dd8-4fad-8291-54d04507d67b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:00.318328 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.318299 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901df951-0dd8-4fad-8291-54d04507d67b-kube-api-access-6kk5w" pod "901df951-0dd8-4fad-8291-54d04507d67b" (UID: "901df951-0dd8-4fad-8291-54d04507d67b"). InnerVolumeSpecName "kube-api-access-6kk5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:00.403987 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.403948 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:00.403987 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.403988 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/901df951-0dd8-4fad-8291-54d04507d67b-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:00.403987 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.404006 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kk5w\" (UniqueName: \"kubernetes.io/projected/901df951-0dd8-4fad-8291-54d04507d67b-kube-api-access-6kk5w\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:00.965877 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.965845 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" Oct 31 00:10:00.966234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.965941 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m7h6k" event={"ID":"901df951-0dd8-4fad-8291-54d04507d67b","Type":"ContainerDied","Data":"823b887387e189a7090cbf49c492d50de942261df6fe4df1da7f35dd16439b17"} Oct 31 00:10:00.966234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:00.965991 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="823b887387e189a7090cbf49c492d50de942261df6fe4df1da7f35dd16439b17" Oct 31 00:10:01.124182 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.124124 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:10:01.211977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.211803 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-bundle\") pod \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " Oct 31 00:10:01.211977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.211854 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-util\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-util\") pod \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " Oct 31 00:10:01.211977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.211916 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-kube-api-access-jlnfl\" (UniqueName: \"kubernetes.io/projected/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-kube-api-access-jlnfl\") pod \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\" (UID: \"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa\") " Oct 31 00:10:01.213392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.213356 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-bundle" pod "cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" (UID: "cc427ff7-e582-4b9d-92bb-8ca28e76fbfa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:01.215330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.215292 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-kube-api-access-jlnfl" pod "cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" (UID: "cc427ff7-e582-4b9d-92bb-8ca28e76fbfa"). InnerVolumeSpecName "kube-api-access-jlnfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:01.220043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.220016 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-util" pod "cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" (UID: "cc427ff7-e582-4b9d-92bb-8ca28e76fbfa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:01.320568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.320478 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.320568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.320512 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.320568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.320527 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlnfl\" (UniqueName: \"kubernetes.io/projected/cc427ff7-e582-4b9d-92bb-8ca28e76fbfa-kube-api-access-jlnfl\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.972401 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.972368 4969 generic.go:356] "Generic (PLEG): container finished" podID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerID="e812ba001761fbf3f1185c2cfac2788461273ff6616c938e17c788e88979bc34" exitCode=0 Oct 31 00:10:01.972888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.972445 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" event={"ID":"8eb27f0e-86c5-4611-a807-eb05397eefd0","Type":"ContainerDied","Data":"e812ba001761fbf3f1185c2cfac2788461273ff6616c938e17c788e88979bc34"} Oct 31 00:10:01.974648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.974624 4969 generic.go:356] "Generic (PLEG): container finished" podID="5571790b-04b2-49d1-93b7-8918952ace4b" containerID="d505c0e2c0f6ea7f64224e16205a602517471fc373f67ba8a92626672a4075a5" exitCode=0 Oct 31 00:10:01.974775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.974675 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" event={"ID":"5571790b-04b2-49d1-93b7-8918952ace4b","Type":"ContainerDied","Data":"d505c0e2c0f6ea7f64224e16205a602517471fc373f67ba8a92626672a4075a5"} Oct 31 00:10:01.976635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.976613 4969 generic.go:356] "Generic (PLEG): container finished" podID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerID="70d227ae8d300d9cf49065c65a24f4f45b864c0bb4efd48c1373dd176bd43a2b" exitCode=0 Oct 31 00:10:01.976724 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.976684 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" event={"ID":"401a3887-09c5-4192-a7df-4c758b19ac8e","Type":"ContainerDied","Data":"70d227ae8d300d9cf49065c65a24f4f45b864c0bb4efd48c1373dd176bd43a2b"} Oct 31 00:10:01.978200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.978183 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:10:01.978868 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.978847 4969 generic.go:356] "Generic (PLEG): container finished" podID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerID="5644495d5404fc643b86dd25e6de3ae3be7daf8e44126b0f2e493ab96a4f8d53" exitCode=1 Oct 31 00:10:01.978949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.978878 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" event={"ID":"fc66fc26-a5d3-4165-8937-b2dae564dab9","Type":"ContainerDied","Data":"5644495d5404fc643b86dd25e6de3ae3be7daf8e44126b0f2e493ab96a4f8d53"} Oct 31 00:10:01.980994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.980977 4969 generic.go:356] "Generic (PLEG): container finished" podID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerID="d18066e1834334bbeb25d770897f8a76f37fe64d57dc2bb426cd2d38b8bc8bb1" exitCode=0 Oct 31 00:10:01.981091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.981042 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" event={"ID":"670c7343-c47d-40b1-9dd1-d7be7a3c7759","Type":"ContainerDied","Data":"d18066e1834334bbeb25d770897f8a76f37fe64d57dc2bb426cd2d38b8bc8bb1"} Oct 31 00:10:01.982857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.982842 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" Oct 31 00:10:01.982857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.982852 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2rvg" event={"ID":"cc427ff7-e582-4b9d-92bb-8ca28e76fbfa","Type":"ContainerDied","Data":"bbae5128ab5aab8bf5aa25d9240a597445522b8f785934bffc81df167fed5122"} Oct 31 00:10:01.983016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.982884 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbae5128ab5aab8bf5aa25d9240a597445522b8f785934bffc81df167fed5122" Oct 31 00:10:01.985302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.985163 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" event={"ID":"4fe5853f-e254-4351-b216-9a2c89a1a306","Type":"ContainerStarted","Data":"45cad0921c8a1be83ab685894e33d8fbd60a15d5f640f4f9bacaa09385d2bc99"} Oct 31 00:10:01.987289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:01.987267 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" event={"ID":"2a71577e-7c5f-4311-a5c7-3ebe07425f6b","Type":"ContainerStarted","Data":"5f9b862812aeafae1a49a23e1e687d632cf0c5b16568a3e3ece8f694d35f80cd"} Oct 31 00:10:02.208858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:02.208754 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" podStartSLOduration=3.14352085 podStartE2EDuration="44.208738737s" totalImagesPullingTime="41.065217887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:02.207133485 +0000 UTC m=+434.682491904" watchObservedRunningTime="2026-10-31 00:10:02.208738737 +0000 UTC m=+434.684097156" Oct 31 00:10:02.558807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:02.558681 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" podStartSLOduration=4.418797949 podStartE2EDuration="46.558660341s" totalImagesPullingTime="42.139862392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:02.558210682 +0000 UTC m=+435.033569105" watchObservedRunningTime="2026-10-31 00:10:02.558660341 +0000 UTC m=+435.034018759" Oct 31 00:10:02.726491 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:10:02.726451 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:10:02.994763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:02.994726 4969 generic.go:356] "Generic (PLEG): container finished" podID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerID="45cad0921c8a1be83ab685894e33d8fbd60a15d5f640f4f9bacaa09385d2bc99" exitCode=0 Oct 31 00:10:02.995192 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:02.994823 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" event={"ID":"4fe5853f-e254-4351-b216-9a2c89a1a306","Type":"ContainerDied","Data":"45cad0921c8a1be83ab685894e33d8fbd60a15d5f640f4f9bacaa09385d2bc99"} Oct 31 00:10:02.996882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:02.996859 4969 generic.go:356] "Generic (PLEG): container finished" podID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerID="5f9b862812aeafae1a49a23e1e687d632cf0c5b16568a3e3ece8f694d35f80cd" exitCode=0 Oct 31 00:10:02.997021 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:02.996950 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" event={"ID":"2a71577e-7c5f-4311-a5c7-3ebe07425f6b","Type":"ContainerDied","Data":"5f9b862812aeafae1a49a23e1e687d632cf0c5b16568a3e3ece8f694d35f80cd"} Oct 31 00:10:03.158868 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.158836 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:10:03.191118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.191087 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:10:03.219556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.219528 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:10:03.221886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.221861 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:10:03.224571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.224548 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:10:03.234240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.234212 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:10:03.237647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237624 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670c7343-c47d-40b1-9dd1-d7be7a3c7759-kube-api-access-wv2wv\" (UniqueName: \"kubernetes.io/projected/670c7343-c47d-40b1-9dd1-d7be7a3c7759-kube-api-access-wv2wv\") pod \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " Oct 31 00:10:03.237769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237659 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-util\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-util\") pod \"5571790b-04b2-49d1-93b7-8918952ace4b\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " Oct 31 00:10:03.237769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237682 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-bundle\") pod \"401a3887-09c5-4192-a7df-4c758b19ac8e\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " Oct 31 00:10:03.237769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237708 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-util\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-util\") pod \"401a3887-09c5-4192-a7df-4c758b19ac8e\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " Oct 31 00:10:03.237769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237727 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-util\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-util\") pod \"fc66fc26-a5d3-4165-8937-b2dae564dab9\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " Oct 31 00:10:03.237769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237753 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc66fc26-a5d3-4165-8937-b2dae564dab9-kube-api-access-4x7dl\" (UniqueName: \"kubernetes.io/projected/fc66fc26-a5d3-4165-8937-b2dae564dab9-kube-api-access-4x7dl\") pod \"fc66fc26-a5d3-4165-8937-b2dae564dab9\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " Oct 31 00:10:03.238019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237784 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-util\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-util\") pod \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " Oct 31 00:10:03.238019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237844 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-util\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-util\") pod \"8eb27f0e-86c5-4611-a807-eb05397eefd0\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " Oct 31 00:10:03.238019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237878 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb27f0e-86c5-4611-a807-eb05397eefd0-kube-api-access-mdqpg\" (UniqueName: \"kubernetes.io/projected/8eb27f0e-86c5-4611-a807-eb05397eefd0-kube-api-access-mdqpg\") pod \"8eb27f0e-86c5-4611-a807-eb05397eefd0\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " Oct 31 00:10:03.238019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237918 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-bundle\") pod \"8eb27f0e-86c5-4611-a807-eb05397eefd0\" (UID: \"8eb27f0e-86c5-4611-a807-eb05397eefd0\") " Oct 31 00:10:03.238019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237961 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-bundle\") pod \"5571790b-04b2-49d1-93b7-8918952ace4b\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " Oct 31 00:10:03.238019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.237998 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-bundle\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-bundle\") pod \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\" (UID: \"670c7343-c47d-40b1-9dd1-d7be7a3c7759\") " Oct 31 00:10:03.238313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.238026 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401a3887-09c5-4192-a7df-4c758b19ac8e-kube-api-access-mqvjx\" (UniqueName: \"kubernetes.io/projected/401a3887-09c5-4192-a7df-4c758b19ac8e-kube-api-access-mqvjx\") pod \"401a3887-09c5-4192-a7df-4c758b19ac8e\" (UID: \"401a3887-09c5-4192-a7df-4c758b19ac8e\") " Oct 31 00:10:03.238313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.238062 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5571790b-04b2-49d1-93b7-8918952ace4b-kube-api-access-r6bqx\" (UniqueName: \"kubernetes.io/projected/5571790b-04b2-49d1-93b7-8918952ace4b-kube-api-access-r6bqx\") pod \"5571790b-04b2-49d1-93b7-8918952ace4b\" (UID: \"5571790b-04b2-49d1-93b7-8918952ace4b\") " Oct 31 00:10:03.238313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.238104 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-bundle\") pod \"fc66fc26-a5d3-4165-8937-b2dae564dab9\" (UID: \"fc66fc26-a5d3-4165-8937-b2dae564dab9\") " Oct 31 00:10:03.238768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.238744 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-bundle" pod "fc66fc26-a5d3-4165-8937-b2dae564dab9" (UID: "fc66fc26-a5d3-4165-8937-b2dae564dab9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.240354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.240323 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670c7343-c47d-40b1-9dd1-d7be7a3c7759-kube-api-access-wv2wv" pod "670c7343-c47d-40b1-9dd1-d7be7a3c7759" (UID: "670c7343-c47d-40b1-9dd1-d7be7a3c7759"). InnerVolumeSpecName "kube-api-access-wv2wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:03.240441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.240375 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc66fc26-a5d3-4165-8937-b2dae564dab9-kube-api-access-4x7dl" pod "fc66fc26-a5d3-4165-8937-b2dae564dab9" (UID: "fc66fc26-a5d3-4165-8937-b2dae564dab9"). InnerVolumeSpecName "kube-api-access-4x7dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:03.240793 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.240768 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-bundle" pod "5571790b-04b2-49d1-93b7-8918952ace4b" (UID: "5571790b-04b2-49d1-93b7-8918952ace4b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.241353 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.241333 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-bundle" pod "8eb27f0e-86c5-4611-a807-eb05397eefd0" (UID: "8eb27f0e-86c5-4611-a807-eb05397eefd0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.241902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.241879 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-bundle" pod "401a3887-09c5-4192-a7df-4c758b19ac8e" (UID: "401a3887-09c5-4192-a7df-4c758b19ac8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.242751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.242724 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb27f0e-86c5-4611-a807-eb05397eefd0-kube-api-access-mdqpg" pod "8eb27f0e-86c5-4611-a807-eb05397eefd0" (UID: "8eb27f0e-86c5-4611-a807-eb05397eefd0"). InnerVolumeSpecName "kube-api-access-mdqpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:03.243390 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.243326 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-bundle" pod "670c7343-c47d-40b1-9dd1-d7be7a3c7759" (UID: "670c7343-c47d-40b1-9dd1-d7be7a3c7759"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.243619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.243566 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401a3887-09c5-4192-a7df-4c758b19ac8e-kube-api-access-mqvjx" pod "401a3887-09c5-4192-a7df-4c758b19ac8e" (UID: "401a3887-09c5-4192-a7df-4c758b19ac8e"). InnerVolumeSpecName "kube-api-access-mqvjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:03.245249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.245186 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5571790b-04b2-49d1-93b7-8918952ace4b-kube-api-access-r6bqx" pod "5571790b-04b2-49d1-93b7-8918952ace4b" (UID: "5571790b-04b2-49d1-93b7-8918952ace4b"). InnerVolumeSpecName "kube-api-access-r6bqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:03.246457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.246430 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-util" pod "5571790b-04b2-49d1-93b7-8918952ace4b" (UID: "5571790b-04b2-49d1-93b7-8918952ace4b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.246833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.246812 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-util" pod "401a3887-09c5-4192-a7df-4c758b19ac8e" (UID: "401a3887-09c5-4192-a7df-4c758b19ac8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.248309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.248282 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-util" pod "670c7343-c47d-40b1-9dd1-d7be7a3c7759" (UID: "670c7343-c47d-40b1-9dd1-d7be7a3c7759"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.249104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.249080 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-util" pod "fc66fc26-a5d3-4165-8937-b2dae564dab9" (UID: "fc66fc26-a5d3-4165-8937-b2dae564dab9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.249176 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.249156 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-util" pod "8eb27f0e-86c5-4611-a807-eb05397eefd0" (UID: "8eb27f0e-86c5-4611-a807-eb05397eefd0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:03.339010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.338968 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339000 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqvjx\" (UniqueName: \"kubernetes.io/projected/401a3887-09c5-4192-a7df-4c758b19ac8e-kube-api-access-mqvjx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339011 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6bqx\" (UniqueName: \"kubernetes.io/projected/5571790b-04b2-49d1-93b7-8918952ace4b-kube-api-access-r6bqx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339020 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339030 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2wv\" (UniqueName: \"kubernetes.io/projected/670c7343-c47d-40b1-9dd1-d7be7a3c7759-kube-api-access-wv2wv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339038 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339047 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339055 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/401a3887-09c5-4192-a7df-4c758b19ac8e-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339062 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc66fc26-a5d3-4165-8937-b2dae564dab9-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339070 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7dl\" (UniqueName: \"kubernetes.io/projected/fc66fc26-a5d3-4165-8937-b2dae564dab9-kube-api-access-4x7dl\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339079 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/670c7343-c47d-40b1-9dd1-d7be7a3c7759-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339087 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339095 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdqpg\" (UniqueName: \"kubernetes.io/projected/8eb27f0e-86c5-4611-a807-eb05397eefd0-kube-api-access-mdqpg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339103 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eb27f0e-86c5-4611-a807-eb05397eefd0-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.339268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:03.339112 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5571790b-04b2-49d1-93b7-8918952ace4b-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:04.003630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.003516 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" event={"ID":"8eb27f0e-86c5-4611-a807-eb05397eefd0","Type":"ContainerDied","Data":"24c4f06d6f61e9f634acf48d723aac3b423553195df24aa7d93b91af308f88ee"} Oct 31 00:10:04.003630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.003609 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24c4f06d6f61e9f634acf48d723aac3b423553195df24aa7d93b91af308f88ee" Oct 31 00:10:04.003630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.003532 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clx8zd" Oct 31 00:10:04.005524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.005494 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" event={"ID":"5571790b-04b2-49d1-93b7-8918952ace4b","Type":"ContainerDied","Data":"6faaca36bd740535283534e19c424e4810a08a2d45f3451b8e9a2f75379a1949"} Oct 31 00:10:04.005669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.005536 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6faaca36bd740535283534e19c424e4810a08a2d45f3451b8e9a2f75379a1949" Oct 31 00:10:04.005669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.005543 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64mrfn" Oct 31 00:10:04.007619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.007580 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" event={"ID":"401a3887-09c5-4192-a7df-4c758b19ac8e","Type":"ContainerDied","Data":"f93d2188e2dc33188b098c959fceefcf266ac5514bfff2ce9639101acee2e4f7"} Oct 31 00:10:04.007716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.007635 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93d2188e2dc33188b098c959fceefcf266ac5514bfff2ce9639101acee2e4f7" Oct 31 00:10:04.007716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.007672 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wccxx" Oct 31 00:10:04.009000 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.008978 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:10:04.009753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.009733 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" Oct 31 00:10:04.009753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.009740 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz" event={"ID":"fc66fc26-a5d3-4165-8937-b2dae564dab9","Type":"ContainerDied","Data":"aec88f72198ec258d3c1a21ee498d37a7c6f450dc5f904e3cd401d3d70ce0344"} Oct 31 00:10:04.009889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.009778 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec88f72198ec258d3c1a21ee498d37a7c6f450dc5f904e3cd401d3d70ce0344" Oct 31 00:10:04.011924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.011907 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" Oct 31 00:10:04.012013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.011932 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfp9s5" event={"ID":"670c7343-c47d-40b1-9dd1-d7be7a3c7759","Type":"ContainerDied","Data":"3f694f73707eedbd4bfae0a790fd5b211ee4d3b11a57f0527ac4684fb77837c0"} Oct 31 00:10:04.012013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.011967 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f694f73707eedbd4bfae0a790fd5b211ee4d3b11a57f0527ac4684fb77837c0" Oct 31 00:10:04.165790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.165770 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:10:04.168084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.168065 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:10:04.244830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.244795 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-util\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-util\") pod \"4fe5853f-e254-4351-b216-9a2c89a1a306\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " Oct 31 00:10:04.244830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.244835 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe5853f-e254-4351-b216-9a2c89a1a306-kube-api-access-wlqwf\" (UniqueName: \"kubernetes.io/projected/4fe5853f-e254-4351-b216-9a2c89a1a306-kube-api-access-wlqwf\") pod \"4fe5853f-e254-4351-b216-9a2c89a1a306\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " Oct 31 00:10:04.245049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.244867 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-util\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-util\") pod \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " Oct 31 00:10:04.245049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.244903 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-bundle\") pod \"4fe5853f-e254-4351-b216-9a2c89a1a306\" (UID: \"4fe5853f-e254-4351-b216-9a2c89a1a306\") " Oct 31 00:10:04.245049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.244937 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-bundle\") pod \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " Oct 31 00:10:04.245049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.244982 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-kube-api-access-ssqkr\" (UniqueName: \"kubernetes.io/projected/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-kube-api-access-ssqkr\") pod \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\" (UID: \"2a71577e-7c5f-4311-a5c7-3ebe07425f6b\") " Oct 31 00:10:04.245383 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.245354 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-bundle" pod "4fe5853f-e254-4351-b216-9a2c89a1a306" (UID: "4fe5853f-e254-4351-b216-9a2c89a1a306"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:04.245746 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.245724 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-bundle" pod "2a71577e-7c5f-4311-a5c7-3ebe07425f6b" (UID: "2a71577e-7c5f-4311-a5c7-3ebe07425f6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:04.247491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.247462 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-kube-api-access-ssqkr" pod "2a71577e-7c5f-4311-a5c7-3ebe07425f6b" (UID: "2a71577e-7c5f-4311-a5c7-3ebe07425f6b"). InnerVolumeSpecName "kube-api-access-ssqkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:04.247672 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.247652 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe5853f-e254-4351-b216-9a2c89a1a306-kube-api-access-wlqwf" pod "4fe5853f-e254-4351-b216-9a2c89a1a306" (UID: "4fe5853f-e254-4351-b216-9a2c89a1a306"). InnerVolumeSpecName "kube-api-access-wlqwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:04.251088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.251062 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-util" pod "2a71577e-7c5f-4311-a5c7-3ebe07425f6b" (UID: "2a71577e-7c5f-4311-a5c7-3ebe07425f6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:04.252012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.251869 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-util" pod "4fe5853f-e254-4351-b216-9a2c89a1a306" (UID: "4fe5853f-e254-4351-b216-9a2c89a1a306"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:04.345522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.345445 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:04.345522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.345472 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:04.345522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.345483 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:04.345522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.345492 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssqkr\" (UniqueName: \"kubernetes.io/projected/2a71577e-7c5f-4311-a5c7-3ebe07425f6b-kube-api-access-ssqkr\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:04.345522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.345501 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4fe5853f-e254-4351-b216-9a2c89a1a306-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:04.345522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:04.345510 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlqwf\" (UniqueName: \"kubernetes.io/projected/4fe5853f-e254-4351-b216-9a2c89a1a306-kube-api-access-wlqwf\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:05.019045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.019007 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" event={"ID":"4fe5853f-e254-4351-b216-9a2c89a1a306","Type":"ContainerDied","Data":"2c887c0487759390cadce040632452689d81251b937aaa702196b679e0593047"} Oct 31 00:10:05.019045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.019031 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6vk2" Oct 31 00:10:05.019515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.019057 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c887c0487759390cadce040632452689d81251b937aaa702196b679e0593047" Oct 31 00:10:05.020912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.020887 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" Oct 31 00:10:05.021043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.020911 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqfjhh" event={"ID":"2a71577e-7c5f-4311-a5c7-3ebe07425f6b","Type":"ContainerDied","Data":"b414a831afcad3bda0c11c1a12ca0959698e0bf3752e170dd30ac99df5d37736"} Oct 31 00:10:05.021043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.020961 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b414a831afcad3bda0c11c1a12ca0959698e0bf3752e170dd30ac99df5d37736" Oct 31 00:10:05.533578 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533546 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b"] Oct 31 00:10:05.533867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533855 4969 cpu_manager.go:414] "containerMap: removing container" podUID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerName="extract" Oct 31 00:10:05.533914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533869 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerName="extract" Oct 31 00:10:05.533914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533879 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerName="util" Oct 31 00:10:05.533914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533884 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerName="util" Oct 31 00:10:05.533914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533893 4969 cpu_manager.go:414] "containerMap: removing container" podUID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerName="util" Oct 31 00:10:05.533914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533898 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerName="util" Oct 31 00:10:05.533914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533906 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerName="pull" Oct 31 00:10:05.533914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533911 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533918 4969 cpu_manager.go:414] "containerMap: removing container" podUID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533922 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533933 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerName="util" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533938 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerName="util" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533946 4969 cpu_manager.go:414] "containerMap: removing container" podUID="5571790b-04b2-49d1-93b7-8918952ace4b" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533951 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5571790b-04b2-49d1-93b7-8918952ace4b" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533956 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533960 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533966 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533971 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533978 4969 cpu_manager.go:414] "containerMap: removing container" podUID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533982 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533988 4969 cpu_manager.go:414] "containerMap: removing container" podUID="901df951-0dd8-4fad-8291-54d04507d67b" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.533992 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901df951-0dd8-4fad-8291-54d04507d67b" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534001 4969 cpu_manager.go:414] "containerMap: removing container" podUID="901df951-0dd8-4fad-8291-54d04507d67b" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534004 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901df951-0dd8-4fad-8291-54d04507d67b" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534011 4969 cpu_manager.go:414] "containerMap: removing container" podUID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534014 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534022 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534026 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534032 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534036 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerName="pull" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534041 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534045 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534053 4969 cpu_manager.go:414] "containerMap: removing container" podUID="5571790b-04b2-49d1-93b7-8918952ace4b" containerName="util" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534056 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5571790b-04b2-49d1-93b7-8918952ace4b" containerName="util" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534062 4969 cpu_manager.go:414] "containerMap: removing container" podUID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534065 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534070 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534074 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534080 4969 cpu_manager.go:414] "containerMap: removing container" podUID="901df951-0dd8-4fad-8291-54d04507d67b" containerName="util" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534085 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901df951-0dd8-4fad-8291-54d04507d67b" containerName="util" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534091 4969 cpu_manager.go:414] "containerMap: removing container" podUID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534095 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534101 4969 cpu_manager.go:414] "containerMap: removing container" podUID="5571790b-04b2-49d1-93b7-8918952ace4b" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534106 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5571790b-04b2-49d1-93b7-8918952ace4b" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534112 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerName="util" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534116 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerName="util" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534123 4969 cpu_manager.go:414] "containerMap: removing container" podUID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534127 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerName="extract" Oct 31 00:10:05.534109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534133 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerName="util" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534137 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerName="util" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534143 4969 cpu_manager.go:414] "containerMap: removing container" podUID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerName="pull" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534148 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerName="pull" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534158 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534164 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534177 4969 cpu_manager.go:414] "containerMap: removing container" podUID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerName="util" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534182 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerName="util" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534191 4969 cpu_manager.go:414] "containerMap: removing container" podUID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerName="util" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534197 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerName="util" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534205 4969 cpu_manager.go:414] "containerMap: removing container" podUID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerName="util" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534211 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerName="util" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534293 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="670c7343-c47d-40b1-9dd1-d7be7a3c7759" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534309 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d86f964-fe4e-417e-928e-56f38b0b84d8" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534319 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="901df951-0dd8-4fad-8291-54d04507d67b" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534329 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb27f0e-86c5-4611-a807-eb05397eefd0" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534337 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc66fc26-a5d3-4165-8937-b2dae564dab9" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534346 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a71577e-7c5f-4311-a5c7-3ebe07425f6b" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534357 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc427ff7-e582-4b9d-92bb-8ca28e76fbfa" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534365 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="5571790b-04b2-49d1-93b7-8918952ace4b" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534375 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="401a3887-09c5-4192-a7df-4c758b19ac8e" containerName="extract" Oct 31 00:10:05.535265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.534380 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe5853f-e254-4351-b216-9a2c89a1a306" containerName="extract" Oct 31 00:10:05.536818 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.536802 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.555123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.555097 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c2a9ebe-2939-4117-b5d6-843887093d61-webhook-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.555240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.555147 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c2a9ebe-2939-4117-b5d6-843887093d61-metrics-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.555240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.555227 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c2a9ebe-2939-4117-b5d6-843887093d61-tmp\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.555369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.555266 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c2a9ebe-2939-4117-b5d6-843887093d61-manifests\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.555369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.555347 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdb69\" (UniqueName: \"kubernetes.io/projected/7c2a9ebe-2939-4117-b5d6-843887093d61-kube-api-access-xdb69\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.555467 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.555389 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c2a9ebe-2939-4117-b5d6-843887093d61-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.595915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.595876 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 00:10:05.596152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.595991 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 00:10:05.596152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.596014 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 00:10:05.596152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.596060 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 00:10:05.596269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.596153 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 00:10:05.596269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.596219 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-25dgf\"" Oct 31 00:10:05.656655 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.656616 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c2a9ebe-2939-4117-b5d6-843887093d61-webhook-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.656802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.656680 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c2a9ebe-2939-4117-b5d6-843887093d61-metrics-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.656802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.656712 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c2a9ebe-2939-4117-b5d6-843887093d61-tmp\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.656802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.656735 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c2a9ebe-2939-4117-b5d6-843887093d61-manifests\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.656802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.656765 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdb69\" (UniqueName: \"kubernetes.io/projected/7c2a9ebe-2939-4117-b5d6-843887093d61-kube-api-access-xdb69\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.656986 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.656808 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c2a9ebe-2939-4117-b5d6-843887093d61-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.657247 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.657209 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7c2a9ebe-2939-4117-b5d6-843887093d61-manifests\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.659239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.659216 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c2a9ebe-2939-4117-b5d6-843887093d61-tmp\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.659478 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.659456 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7c2a9ebe-2939-4117-b5d6-843887093d61-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.659575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.659513 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c2a9ebe-2939-4117-b5d6-843887093d61-metrics-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.659668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.659631 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c2a9ebe-2939-4117-b5d6-843887093d61-webhook-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.731661 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.731630 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdb69\" (UniqueName: \"kubernetes.io/projected/7c2a9ebe-2939-4117-b5d6-843887093d61-kube-api-access-xdb69\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-r572b\" (UID: \"7c2a9ebe-2939-4117-b5d6-843887093d61\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:05.814627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.814528 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b"] Oct 31 00:10:05.848770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:05.848731 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:06.337919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:06.337891 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b"] Oct 31 00:10:06.340509 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:06.340478 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2a9ebe_2939_4117_b5d6_843887093d61.slice/crio-d61340374da24656cadf7bafcfb051653ef260911b9fd53eb635cc182d3e7be2 WatchSource:0}: Error finding container d61340374da24656cadf7bafcfb051653ef260911b9fd53eb635cc182d3e7be2: Status 404 returned error can't find the container with id d61340374da24656cadf7bafcfb051653ef260911b9fd53eb635cc182d3e7be2 Oct 31 00:10:07.032671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:07.032632 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" event={"ID":"7c2a9ebe-2939-4117-b5d6-843887093d61","Type":"ContainerStarted","Data":"d61340374da24656cadf7bafcfb051653ef260911b9fd53eb635cc182d3e7be2"} Oct 31 00:10:10.050706 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:10.050670 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" event={"ID":"7c2a9ebe-2939-4117-b5d6-843887093d61","Type":"ContainerStarted","Data":"64149399d45d4a4872b51a13e24afbda656c1d4e1220a960305e10814217a44b"} Oct 31 00:10:11.056810 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.056775 4969 generic.go:356] "Generic (PLEG): container finished" podID="7c2a9ebe-2939-4117-b5d6-843887093d61" containerID="64149399d45d4a4872b51a13e24afbda656c1d4e1220a960305e10814217a44b" exitCode=0 Oct 31 00:10:11.056810 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.056814 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" event={"ID":"7c2a9ebe-2939-4117-b5d6-843887093d61","Type":"ContainerDied","Data":"64149399d45d4a4872b51a13e24afbda656c1d4e1220a960305e10814217a44b"} Oct 31 00:10:11.068520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.068485 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85"] Oct 31 00:10:11.070732 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.070717 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.100095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.100058 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8jnd\" (UniqueName: \"kubernetes.io/projected/a2637bec-928b-4d9c-9e60-d578f49e4d74-kube-api-access-n8jnd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.100252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.100104 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.100252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.100230 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.178562 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.178527 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:10:11.178755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.178617 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:10:11.178755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.178659 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2x5zb\"" Oct 31 00:10:11.180409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.180391 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85"] Oct 31 00:10:11.201115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.201092 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8jnd\" (UniqueName: \"kubernetes.io/projected/a2637bec-928b-4d9c-9e60-d578f49e4d74-kube-api-access-n8jnd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.201214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.201122 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.201214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.201180 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.201501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.201487 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.201562 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.201544 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.262449 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.262420 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8jnd\" (UniqueName: \"kubernetes.io/projected/a2637bec-928b-4d9c-9e60-d578f49e4d74-kube-api-access-n8jnd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.382772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.382728 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:11.748490 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.748399 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4"] Oct 31 00:10:11.751381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.751357 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" Oct 31 00:10:11.791976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.791958 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85"] Oct 31 00:10:11.793528 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:11.793505 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2637bec_928b_4d9c_9e60_d578f49e4d74.slice/crio-c72b529624706bbbcd42ec916ad9fe38c538138e8af9e8d76ff3719c39d885ac WatchSource:0}: Error finding container c72b529624706bbbcd42ec916ad9fe38c538138e8af9e8d76ff3719c39d885ac: Status 404 returned error can't find the container with id c72b529624706bbbcd42ec916ad9fe38c538138e8af9e8d76ff3719c39d885ac Oct 31 00:10:11.807343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.807313 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlv6c\" (UniqueName: \"kubernetes.io/projected/55141f91-08ab-43b1-abc0-cc8b239a920e-kube-api-access-vlv6c\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-j98d4\" (UID: \"55141f91-08ab-43b1-abc0-cc8b239a920e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" Oct 31 00:10:11.807459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.807441 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55141f91-08ab-43b1-abc0-cc8b239a920e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-j98d4\" (UID: \"55141f91-08ab-43b1-abc0-cc8b239a920e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" Oct 31 00:10:11.819211 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.819177 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4m6np\"" Oct 31 00:10:11.819318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.819232 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 00:10:11.819318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.819273 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:10:11.829118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.829092 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4"] Oct 31 00:10:11.908636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.908612 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlv6c\" (UniqueName: \"kubernetes.io/projected/55141f91-08ab-43b1-abc0-cc8b239a920e-kube-api-access-vlv6c\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-j98d4\" (UID: \"55141f91-08ab-43b1-abc0-cc8b239a920e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" Oct 31 00:10:11.908742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.908678 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55141f91-08ab-43b1-abc0-cc8b239a920e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-j98d4\" (UID: \"55141f91-08ab-43b1-abc0-cc8b239a920e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" Oct 31 00:10:11.908997 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.908980 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55141f91-08ab-43b1-abc0-cc8b239a920e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-j98d4\" (UID: \"55141f91-08ab-43b1-abc0-cc8b239a920e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" Oct 31 00:10:11.984266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:11.984241 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlv6c\" (UniqueName: \"kubernetes.io/projected/55141f91-08ab-43b1-abc0-cc8b239a920e-kube-api-access-vlv6c\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-j98d4\" (UID: \"55141f91-08ab-43b1-abc0-cc8b239a920e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" Oct 31 00:10:12.061678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:12.061647 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" Oct 31 00:10:12.063971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:12.063798 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" event={"ID":"7c2a9ebe-2939-4117-b5d6-843887093d61","Type":"ContainerStarted","Data":"702b96627b42ab285ce8c6828e739a63847fce94ae4c6d900a4e20dd2722a97b"} Oct 31 00:10:12.063971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:12.063863 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:12.065510 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:12.065481 4969 generic.go:356] "Generic (PLEG): container finished" podID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerID="d8d72ec9f805a31410660b42bb445592367561b0d38361c2f7cf29c68df80949" exitCode=0 Oct 31 00:10:12.065641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:12.065534 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" event={"ID":"a2637bec-928b-4d9c-9e60-d578f49e4d74","Type":"ContainerDied","Data":"d8d72ec9f805a31410660b42bb445592367561b0d38361c2f7cf29c68df80949"} Oct 31 00:10:12.065641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:12.065583 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" event={"ID":"a2637bec-928b-4d9c-9e60-d578f49e4d74","Type":"ContainerStarted","Data":"c72b529624706bbbcd42ec916ad9fe38c538138e8af9e8d76ff3719c39d885ac"} Oct 31 00:10:12.219474 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:12.219417 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" podStartSLOduration=2.421306754 podStartE2EDuration="7.219403848s" totalImagesPullingTime="4.348168805s" totalInitContainerRuntime="449.928289ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:12.216313567 +0000 UTC m=+444.691671989" watchObservedRunningTime="2026-10-31 00:10:12.219403848 +0000 UTC m=+444.694762270" Oct 31 00:10:12.527639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:12.527608 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4"] Oct 31 00:10:12.528246 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:12.528218 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55141f91_08ab_43b1_abc0_cc8b239a920e.slice/crio-b82357b63bc34cc0c72ef7b56132c9a126117c9cd5f5460b5b3d139b0ac3f8d7 WatchSource:0}: Error finding container b82357b63bc34cc0c72ef7b56132c9a126117c9cd5f5460b5b3d139b0ac3f8d7: Status 404 returned error can't find the container with id b82357b63bc34cc0c72ef7b56132c9a126117c9cd5f5460b5b3d139b0ac3f8d7 Oct 31 00:10:13.071384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:13.071349 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" event={"ID":"55141f91-08ab-43b1-abc0-cc8b239a920e","Type":"ContainerStarted","Data":"b82357b63bc34cc0c72ef7b56132c9a126117c9cd5f5460b5b3d139b0ac3f8d7"} Oct 31 00:10:13.073197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:13.073171 4969 generic.go:356] "Generic (PLEG): container finished" podID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerID="0155357b7e185a50068ebaf921293e994e5093c624d42aa1849f34ac839145bf" exitCode=0 Oct 31 00:10:13.073329 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:13.073260 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" event={"ID":"a2637bec-928b-4d9c-9e60-d578f49e4d74","Type":"ContainerDied","Data":"0155357b7e185a50068ebaf921293e994e5093c624d42aa1849f34ac839145bf"} Oct 31 00:10:14.079984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:14.079950 4969 generic.go:356] "Generic (PLEG): container finished" podID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerID="55376601e75a998389853603556a096d30e9caa9cc4bf1a30a19250c3b21b9dc" exitCode=0 Oct 31 00:10:14.080361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:14.080031 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" event={"ID":"a2637bec-928b-4d9c-9e60-d578f49e4d74","Type":"ContainerDied","Data":"55376601e75a998389853603556a096d30e9caa9cc4bf1a30a19250c3b21b9dc"} Oct 31 00:10:15.222196 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.222169 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:15.335161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.335122 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-bundle\") pod \"a2637bec-928b-4d9c-9e60-d578f49e4d74\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " Oct 31 00:10:15.335337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.335198 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2637bec-928b-4d9c-9e60-d578f49e4d74-kube-api-access-n8jnd\" (UniqueName: \"kubernetes.io/projected/a2637bec-928b-4d9c-9e60-d578f49e4d74-kube-api-access-n8jnd\") pod \"a2637bec-928b-4d9c-9e60-d578f49e4d74\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " Oct 31 00:10:15.335337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.335231 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-util\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-util\") pod \"a2637bec-928b-4d9c-9e60-d578f49e4d74\" (UID: \"a2637bec-928b-4d9c-9e60-d578f49e4d74\") " Oct 31 00:10:15.335801 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.335767 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-bundle" pod "a2637bec-928b-4d9c-9e60-d578f49e4d74" (UID: "a2637bec-928b-4d9c-9e60-d578f49e4d74"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:15.338800 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.338737 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2637bec-928b-4d9c-9e60-d578f49e4d74-kube-api-access-n8jnd" pod "a2637bec-928b-4d9c-9e60-d578f49e4d74" (UID: "a2637bec-928b-4d9c-9e60-d578f49e4d74"). InnerVolumeSpecName "kube-api-access-n8jnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:15.345445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.345415 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-util" pod "a2637bec-928b-4d9c-9e60-d578f49e4d74" (UID: "a2637bec-928b-4d9c-9e60-d578f49e4d74"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:15.436639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.436616 4969 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:15.436722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.436643 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8jnd\" (UniqueName: \"kubernetes.io/projected/a2637bec-928b-4d9c-9e60-d578f49e4d74-kube-api-access-n8jnd\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:15.436722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:15.436653 4969 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2637bec-928b-4d9c-9e60-d578f49e4d74-util\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:10:16.093210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:16.093123 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" Oct 31 00:10:16.093210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:16.093142 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s6k85" event={"ID":"a2637bec-928b-4d9c-9e60-d578f49e4d74","Type":"ContainerDied","Data":"c72b529624706bbbcd42ec916ad9fe38c538138e8af9e8d76ff3719c39d885ac"} Oct 31 00:10:16.093210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:16.093192 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c72b529624706bbbcd42ec916ad9fe38c538138e8af9e8d76ff3719c39d885ac" Oct 31 00:10:16.095063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:16.095034 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" event={"ID":"55141f91-08ab-43b1-abc0-cc8b239a920e","Type":"ContainerStarted","Data":"88d2e640f9322c0d18bb068123e4473beb8fe121084858e34c7f95af31bb8554"} Oct 31 00:10:16.317431 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:16.317372 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-j98d4" podStartSLOduration=2.453986787 podStartE2EDuration="5.317356507s" totalImagesPullingTime="2.86336972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:16.267261467 +0000 UTC m=+448.742619889" watchObservedRunningTime="2026-10-31 00:10:16.317356507 +0000 UTC m=+448.792714930" Oct 31 00:10:18.636098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.636065 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm"] Oct 31 00:10:18.636464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.636363 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerName="extract" Oct 31 00:10:18.636464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.636373 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerName="extract" Oct 31 00:10:18.636464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.636388 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerName="pull" Oct 31 00:10:18.636464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.636393 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerName="pull" Oct 31 00:10:18.636464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.636400 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerName="util" Oct 31 00:10:18.636464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.636405 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerName="util" Oct 31 00:10:18.636464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.636466 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2637bec-928b-4d9c-9e60-d578f49e4d74" containerName="extract" Oct 31 00:10:18.639063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.639048 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" Oct 31 00:10:18.664423 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.664404 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddqn4\" (UniqueName: \"kubernetes.io/projected/e25c639d-a7ea-4f32-87b6-a63d6840d169-kube-api-access-ddqn4\") pod \"jobset-operator-674cd59dc5-djnzm\" (UID: \"e25c639d-a7ea-4f32-87b6-a63d6840d169\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" Oct 31 00:10:18.664532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.664437 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25c639d-a7ea-4f32-87b6-a63d6840d169-tmp\") pod \"jobset-operator-674cd59dc5-djnzm\" (UID: \"e25c639d-a7ea-4f32-87b6-a63d6840d169\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" Oct 31 00:10:18.743773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.743736 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 00:10:18.743773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.743763 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-77zqp\"" Oct 31 00:10:18.743944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.743785 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:10:18.745710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.745692 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm"] Oct 31 00:10:18.765240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.765213 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddqn4\" (UniqueName: \"kubernetes.io/projected/e25c639d-a7ea-4f32-87b6-a63d6840d169-kube-api-access-ddqn4\") pod \"jobset-operator-674cd59dc5-djnzm\" (UID: \"e25c639d-a7ea-4f32-87b6-a63d6840d169\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" Oct 31 00:10:18.765354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.765248 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25c639d-a7ea-4f32-87b6-a63d6840d169-tmp\") pod \"jobset-operator-674cd59dc5-djnzm\" (UID: \"e25c639d-a7ea-4f32-87b6-a63d6840d169\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" Oct 31 00:10:18.765584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.765570 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25c639d-a7ea-4f32-87b6-a63d6840d169-tmp\") pod \"jobset-operator-674cd59dc5-djnzm\" (UID: \"e25c639d-a7ea-4f32-87b6-a63d6840d169\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" Oct 31 00:10:18.836357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.836325 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddqn4\" (UniqueName: \"kubernetes.io/projected/e25c639d-a7ea-4f32-87b6-a63d6840d169-kube-api-access-ddqn4\") pod \"jobset-operator-674cd59dc5-djnzm\" (UID: \"e25c639d-a7ea-4f32-87b6-a63d6840d169\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" Oct 31 00:10:18.960825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:18.960751 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" Oct 31 00:10:19.367658 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:19.367632 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm"] Oct 31 00:10:19.369270 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:19.369237 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode25c639d_a7ea_4f32_87b6_a63d6840d169.slice/crio-0c3e4764dcc92d0339920d54e6a0d701a5fbab234086804c24b247c7919ac931 WatchSource:0}: Error finding container 0c3e4764dcc92d0339920d54e6a0d701a5fbab234086804c24b247c7919ac931: Status 404 returned error can't find the container with id 0c3e4764dcc92d0339920d54e6a0d701a5fbab234086804c24b247c7919ac931 Oct 31 00:10:20.122241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:20.122207 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" event={"ID":"e25c639d-a7ea-4f32-87b6-a63d6840d169","Type":"ContainerStarted","Data":"0c3e4764dcc92d0339920d54e6a0d701a5fbab234086804c24b247c7919ac931"} Oct 31 00:10:23.076123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:23.076098 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-r572b" Oct 31 00:10:23.140239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:23.140203 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" event={"ID":"e25c639d-a7ea-4f32-87b6-a63d6840d169","Type":"ContainerStarted","Data":"c7da373f840dfe8502784069011dd2d97fe9085fb212780e8a759f196c19bff3"} Oct 31 00:10:23.447808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:23.447752 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-djnzm" podStartSLOduration=2.519818841 podStartE2EDuration="5.447738254s" totalImagesPullingTime="2.927919413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:23.447255765 +0000 UTC m=+455.922614201" watchObservedRunningTime="2026-10-31 00:10:23.447738254 +0000 UTC m=+455.923096676" Oct 31 00:10:27.763039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.763005 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-rbh5p"] Oct 31 00:10:27.766549 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.766532 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rbh5p" Oct 31 00:10:27.823890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.823863 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rbh5p"] Oct 31 00:10:27.841064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.841033 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49b7l\" (UniqueName: \"kubernetes.io/projected/f4d2dc87-688e-4a60-86f1-c3ec159fd5c7-kube-api-access-49b7l\") pod \"cert-manager-597b95cb84-rbh5p\" (UID: \"f4d2dc87-688e-4a60-86f1-c3ec159fd5c7\") " pod="cert-manager/cert-manager-597b95cb84-rbh5p" Oct 31 00:10:27.841064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.841062 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4d2dc87-688e-4a60-86f1-c3ec159fd5c7-bound-sa-token\") pod \"cert-manager-597b95cb84-rbh5p\" (UID: \"f4d2dc87-688e-4a60-86f1-c3ec159fd5c7\") " pod="cert-manager/cert-manager-597b95cb84-rbh5p" Oct 31 00:10:27.870997 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.870968 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 00:10:27.871105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.871017 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 00:10:27.871257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.871245 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-f4nzk\"" Oct 31 00:10:27.941807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.941771 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49b7l\" (UniqueName: \"kubernetes.io/projected/f4d2dc87-688e-4a60-86f1-c3ec159fd5c7-kube-api-access-49b7l\") pod \"cert-manager-597b95cb84-rbh5p\" (UID: \"f4d2dc87-688e-4a60-86f1-c3ec159fd5c7\") " pod="cert-manager/cert-manager-597b95cb84-rbh5p" Oct 31 00:10:27.941807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:27.941809 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4d2dc87-688e-4a60-86f1-c3ec159fd5c7-bound-sa-token\") pod \"cert-manager-597b95cb84-rbh5p\" (UID: \"f4d2dc87-688e-4a60-86f1-c3ec159fd5c7\") " pod="cert-manager/cert-manager-597b95cb84-rbh5p" Oct 31 00:10:28.002948 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:28.002910 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4d2dc87-688e-4a60-86f1-c3ec159fd5c7-bound-sa-token\") pod \"cert-manager-597b95cb84-rbh5p\" (UID: \"f4d2dc87-688e-4a60-86f1-c3ec159fd5c7\") " pod="cert-manager/cert-manager-597b95cb84-rbh5p" Oct 31 00:10:28.003129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:28.003108 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49b7l\" (UniqueName: \"kubernetes.io/projected/f4d2dc87-688e-4a60-86f1-c3ec159fd5c7-kube-api-access-49b7l\") pod \"cert-manager-597b95cb84-rbh5p\" (UID: \"f4d2dc87-688e-4a60-86f1-c3ec159fd5c7\") " pod="cert-manager/cert-manager-597b95cb84-rbh5p" Oct 31 00:10:28.078450 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:28.078369 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rbh5p" Oct 31 00:10:28.449100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:28.449038 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rbh5p"] Oct 31 00:10:28.451546 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:28.451516 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4d2dc87_688e_4a60_86f1_c3ec159fd5c7.slice/crio-d73a02e701ebb44193f33fe015c3f68084e786589ae98f196f3b92079f6913f6 WatchSource:0}: Error finding container d73a02e701ebb44193f33fe015c3f68084e786589ae98f196f3b92079f6913f6: Status 404 returned error can't find the container with id d73a02e701ebb44193f33fe015c3f68084e786589ae98f196f3b92079f6913f6 Oct 31 00:10:29.172641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:29.172549 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rbh5p" event={"ID":"f4d2dc87-688e-4a60-86f1-c3ec159fd5c7","Type":"ContainerStarted","Data":"d73a02e701ebb44193f33fe015c3f68084e786589ae98f196f3b92079f6913f6"} Oct 31 00:10:30.278411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.278380 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf"] Oct 31 00:10:30.281910 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.281886 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:30.337354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.337326 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf"] Oct 31 00:10:30.362316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.362286 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlr9q\" (UniqueName: \"kubernetes.io/projected/dec2cccd-e151-48f1-b2c5-ec2960453d80-kube-api-access-xlr9q\") pod \"kuadrant-operator-controller-manager-cf8469d7b-csnwf\" (UID: \"dec2cccd-e151-48f1-b2c5-ec2960453d80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:30.362447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.362322 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec2cccd-e151-48f1-b2c5-ec2960453d80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-csnwf\" (UID: \"dec2cccd-e151-48f1-b2c5-ec2960453d80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:30.384157 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.384129 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:10:30.384157 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.384129 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:10:30.384157 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.384156 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m8lzx\"" Oct 31 00:10:30.463544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.463517 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlr9q\" (UniqueName: \"kubernetes.io/projected/dec2cccd-e151-48f1-b2c5-ec2960453d80-kube-api-access-xlr9q\") pod \"kuadrant-operator-controller-manager-cf8469d7b-csnwf\" (UID: \"dec2cccd-e151-48f1-b2c5-ec2960453d80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:30.463544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.463550 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec2cccd-e151-48f1-b2c5-ec2960453d80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-csnwf\" (UID: \"dec2cccd-e151-48f1-b2c5-ec2960453d80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:30.463938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.463920 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec2cccd-e151-48f1-b2c5-ec2960453d80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-csnwf\" (UID: \"dec2cccd-e151-48f1-b2c5-ec2960453d80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:30.528134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.528105 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlr9q\" (UniqueName: \"kubernetes.io/projected/dec2cccd-e151-48f1-b2c5-ec2960453d80-kube-api-access-xlr9q\") pod \"kuadrant-operator-controller-manager-cf8469d7b-csnwf\" (UID: \"dec2cccd-e151-48f1-b2c5-ec2960453d80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:30.594911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:30.594841 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:31.087448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:31.087421 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf"] Oct 31 00:10:31.089628 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:31.089576 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec2cccd_e151_48f1_b2c5_ec2960453d80.slice/crio-9d7b2a4c4d44db689d70ce3a92eef89e7a320f8bd4d28833b310d326ab8633f1 WatchSource:0}: Error finding container 9d7b2a4c4d44db689d70ce3a92eef89e7a320f8bd4d28833b310d326ab8633f1: Status 404 returned error can't find the container with id 9d7b2a4c4d44db689d70ce3a92eef89e7a320f8bd4d28833b310d326ab8633f1 Oct 31 00:10:31.185073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:31.185036 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" event={"ID":"dec2cccd-e151-48f1-b2c5-ec2960453d80","Type":"ContainerStarted","Data":"9d7b2a4c4d44db689d70ce3a92eef89e7a320f8bd4d28833b310d326ab8633f1"} Oct 31 00:10:32.729150 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:10:32.729094 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:10:32.800162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:32.800128 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc"] Oct 31 00:10:32.803490 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:32.803475 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" Oct 31 00:10:32.857784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:32.857755 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wf9mg\"" Oct 31 00:10:32.883767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:32.883700 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74lql\" (UniqueName: \"kubernetes.io/projected/40085cb1-46da-478b-bf95-6da06fd5e85f-kube-api-access-74lql\") pod \"authorino-operator-7b7bb4bc77-b5nfc\" (UID: \"40085cb1-46da-478b-bf95-6da06fd5e85f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" Oct 31 00:10:32.900717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:32.900684 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc"] Oct 31 00:10:32.984574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:32.984535 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lql\" (UniqueName: \"kubernetes.io/projected/40085cb1-46da-478b-bf95-6da06fd5e85f-kube-api-access-74lql\") pod \"authorino-operator-7b7bb4bc77-b5nfc\" (UID: \"40085cb1-46da-478b-bf95-6da06fd5e85f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" Oct 31 00:10:33.050348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:33.050312 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74lql\" (UniqueName: \"kubernetes.io/projected/40085cb1-46da-478b-bf95-6da06fd5e85f-kube-api-access-74lql\") pod \"authorino-operator-7b7bb4bc77-b5nfc\" (UID: \"40085cb1-46da-478b-bf95-6da06fd5e85f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" Oct 31 00:10:33.115239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:33.115207 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" Oct 31 00:10:33.208009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:33.207964 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rbh5p" event={"ID":"f4d2dc87-688e-4a60-86f1-c3ec159fd5c7","Type":"ContainerStarted","Data":"85f38ec2c761ee20ebfdcefffb4c5ea077fa7ac8f1df1ac2cc7942fb36380f0e"} Oct 31 00:10:33.328018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:33.327956 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-rbh5p" podStartSLOduration=2.151484389 podStartE2EDuration="6.327937248s" totalImagesPullingTime="4.176452859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:33.325883625 +0000 UTC m=+465.801242049" watchObservedRunningTime="2026-10-31 00:10:33.327937248 +0000 UTC m=+465.803295672" Oct 31 00:10:33.508868 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:33.508846 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc"] Oct 31 00:10:33.511748 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:33.511719 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40085cb1_46da_478b_bf95_6da06fd5e85f.slice/crio-c0bc06d45443258ac5288a930fcd19847b9f8b5e0b83228eab7ba62982e8a874 WatchSource:0}: Error finding container c0bc06d45443258ac5288a930fcd19847b9f8b5e0b83228eab7ba62982e8a874: Status 404 returned error can't find the container with id c0bc06d45443258ac5288a930fcd19847b9f8b5e0b83228eab7ba62982e8a874 Oct 31 00:10:34.216731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:34.216698 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" event={"ID":"40085cb1-46da-478b-bf95-6da06fd5e85f","Type":"ContainerStarted","Data":"c0bc06d45443258ac5288a930fcd19847b9f8b5e0b83228eab7ba62982e8a874"} Oct 31 00:10:37.234506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:37.234462 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" event={"ID":"dec2cccd-e151-48f1-b2c5-ec2960453d80","Type":"ContainerStarted","Data":"435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a"} Oct 31 00:10:37.235008 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:37.234656 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:37.236177 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:37.236152 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" event={"ID":"40085cb1-46da-478b-bf95-6da06fd5e85f","Type":"ContainerStarted","Data":"67250d2677d792bfad08f464ce3b4a0d7b38f50521961fc8ddf00b08de261ad7"} Oct 31 00:10:37.236318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:37.236274 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" Oct 31 00:10:37.456193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:37.456129 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" podStartSLOduration=1.753106799 podStartE2EDuration="7.456110527s" totalImagesPullingTime="5.703003728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:37.403625347 +0000 UTC m=+469.878983763" watchObservedRunningTime="2026-10-31 00:10:37.456110527 +0000 UTC m=+469.931468951" Oct 31 00:10:37.524373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:37.524269 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" podStartSLOduration=2.417781876 podStartE2EDuration="5.524249775s" totalImagesPullingTime="3.106467899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:37.522260697 +0000 UTC m=+469.997619120" watchObservedRunningTime="2026-10-31 00:10:37.524249775 +0000 UTC m=+469.999608202" Oct 31 00:10:37.975579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:37.975540 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m"] Oct 31 00:10:37.979456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:37.979424 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" Oct 31 00:10:38.030925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:38.030891 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cp88\" (UniqueName: \"kubernetes.io/projected/7e3ba2f7-5e86-496b-82fb-956ac8db9d60-kube-api-access-5cp88\") pod \"dns-operator-controller-manager-8557cbdb84-zsk4m\" (UID: \"7e3ba2f7-5e86-496b-82fb-956ac8db9d60\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" Oct 31 00:10:38.032557 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:38.032538 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 00:10:38.049974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:38.049937 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m"] Oct 31 00:10:38.058384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:38.058364 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bnvw7\"" Oct 31 00:10:38.132061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:38.132028 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cp88\" (UniqueName: \"kubernetes.io/projected/7e3ba2f7-5e86-496b-82fb-956ac8db9d60-kube-api-access-5cp88\") pod \"dns-operator-controller-manager-8557cbdb84-zsk4m\" (UID: \"7e3ba2f7-5e86-496b-82fb-956ac8db9d60\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" Oct 31 00:10:38.207500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:38.207470 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cp88\" (UniqueName: \"kubernetes.io/projected/7e3ba2f7-5e86-496b-82fb-956ac8db9d60-kube-api-access-5cp88\") pod \"dns-operator-controller-manager-8557cbdb84-zsk4m\" (UID: \"7e3ba2f7-5e86-496b-82fb-956ac8db9d60\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" Oct 31 00:10:38.292763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:38.292690 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" Oct 31 00:10:38.682963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:38.682796 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m"] Oct 31 00:10:38.685643 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:38.685584 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3ba2f7_5e86_496b_82fb_956ac8db9d60.slice/crio-d132a8886dd223b79e6e6ba72035ce5dfa36d35b77d3cb5a8feb0f82693eaf29 WatchSource:0}: Error finding container d132a8886dd223b79e6e6ba72035ce5dfa36d35b77d3cb5a8feb0f82693eaf29: Status 404 returned error can't find the container with id d132a8886dd223b79e6e6ba72035ce5dfa36d35b77d3cb5a8feb0f82693eaf29 Oct 31 00:10:39.247412 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.247379 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" event={"ID":"7e3ba2f7-5e86-496b-82fb-956ac8db9d60","Type":"ContainerStarted","Data":"d132a8886dd223b79e6e6ba72035ce5dfa36d35b77d3cb5a8feb0f82693eaf29"} Oct 31 00:10:39.855642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.855602 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6"] Oct 31 00:10:39.860345 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.860321 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.917330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.917304 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-x77qk\"" Oct 31 00:10:39.945995 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.945963 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.946130 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.946005 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.946130 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.946091 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.946130 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.946126 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.946297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.946147 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwlf8\" (UniqueName: \"kubernetes.io/projected/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-kube-api-access-xwlf8\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.946297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.946260 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.946387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.946319 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.946421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.946378 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.946421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.946411 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:39.967731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:39.967698 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6"] Oct 31 00:10:40.047567 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.047539 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.047567 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.047571 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.047797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.047619 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwlf8\" (UniqueName: \"kubernetes.io/projected/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-kube-api-access-xwlf8\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.047797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.047648 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.047797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.047679 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.047797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.047703 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.047797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.047721 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.047797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.047748 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.047797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.047781 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.048201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.048092 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.048201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.048176 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.048401 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.048380 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.048484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.048441 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.048484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.048453 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.049938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.049917 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.050280 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.050259 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.108817 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.108749 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.108956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.108927 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwlf8\" (UniqueName: \"kubernetes.io/projected/3dad313d-11a2-440b-a9f9-c0509d6bbcc6-kube-api-access-xwlf8\") pod \"data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6\" (UID: \"3dad313d-11a2-440b-a9f9-c0509d6bbcc6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.174869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.174847 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:40.610152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.610120 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6"] Oct 31 00:10:40.611947 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:40.611919 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dad313d_11a2_440b_a9f9_c0509d6bbcc6.slice/crio-7fb91b73cef4ea5ff876b7c1f887c9d0ad10139b8757b67c6c4d118b0261414b WatchSource:0}: Error finding container 7fb91b73cef4ea5ff876b7c1f887c9d0ad10139b8757b67c6c4d118b0261414b: Status 404 returned error can't find the container with id 7fb91b73cef4ea5ff876b7c1f887c9d0ad10139b8757b67c6c4d118b0261414b Oct 31 00:10:40.614335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.614303 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:10:40.614444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.614384 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:10:40.614444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:40.614428 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:10:41.264444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:41.264402 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" event={"ID":"3dad313d-11a2-440b-a9f9-c0509d6bbcc6","Type":"ContainerStarted","Data":"5b94fff019788327074ae82a2cec7a32cc04be355eaf0bdb392d66ed70c5979b"} Oct 31 00:10:41.264961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:41.264469 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" event={"ID":"3dad313d-11a2-440b-a9f9-c0509d6bbcc6","Type":"ContainerStarted","Data":"7fb91b73cef4ea5ff876b7c1f887c9d0ad10139b8757b67c6c4d118b0261414b"} Oct 31 00:10:41.387028 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:41.386974 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" podStartSLOduration=2.3869602580000002 podStartE2EDuration="2.386960258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:41.384330399 +0000 UTC m=+473.859688822" watchObservedRunningTime="2026-10-31 00:10:41.386960258 +0000 UTC m=+473.862318735" Oct 31 00:10:42.175774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:42.175744 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:42.180652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:42.180629 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:42.271123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:42.271090 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" event={"ID":"7e3ba2f7-5e86-496b-82fb-956ac8db9d60","Type":"ContainerStarted","Data":"5ed0ba427a887e2ba798ca79c5cb1d530fc3dad55c44d4239966ad50cef87556"} Oct 31 00:10:42.271634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:42.271152 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" Oct 31 00:10:42.271634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:42.271187 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:42.272150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:42.272133 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6" Oct 31 00:10:42.578292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:42.578190 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" podStartSLOduration=2.820669271 podStartE2EDuration="5.578175331s" totalImagesPullingTime="2.75750606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:42.576254265 +0000 UTC m=+475.051612689" watchObservedRunningTime="2026-10-31 00:10:42.578175331 +0000 UTC m=+475.053533754" Oct 31 00:10:43.691120 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.691087 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gc9gd"] Oct 31 00:10:43.704403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.704374 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:43.707546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.707522 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gc9gd"] Oct 31 00:10:43.756904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.756875 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 00:10:43.756904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.756889 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kjrmg\"" Oct 31 00:10:43.783282 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.783255 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2cmp\" (UniqueName: \"kubernetes.io/projected/3071e11c-bd77-4a11-bccf-f7635ec38ed9-kube-api-access-p2cmp\") pod \"limitador-limitador-74dfcfb894-gc9gd\" (UID: \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:43.783355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.783324 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3071e11c-bd77-4a11-bccf-f7635ec38ed9-config-file\") pod \"limitador-limitador-74dfcfb894-gc9gd\" (UID: \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:43.884450 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.884417 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3071e11c-bd77-4a11-bccf-f7635ec38ed9-config-file\") pod \"limitador-limitador-74dfcfb894-gc9gd\" (UID: \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:43.884639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.884488 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2cmp\" (UniqueName: \"kubernetes.io/projected/3071e11c-bd77-4a11-bccf-f7635ec38ed9-kube-api-access-p2cmp\") pod \"limitador-limitador-74dfcfb894-gc9gd\" (UID: \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:43.885071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.885049 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3071e11c-bd77-4a11-bccf-f7635ec38ed9-config-file\") pod \"limitador-limitador-74dfcfb894-gc9gd\" (UID: \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:43.890290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.890262 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gc9gd"] Oct 31 00:10:43.945563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:43.945497 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2cmp\" (UniqueName: \"kubernetes.io/projected/3071e11c-bd77-4a11-bccf-f7635ec38ed9-kube-api-access-p2cmp\") pod \"limitador-limitador-74dfcfb894-gc9gd\" (UID: \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:44.035273 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:44.035243 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:44.431303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:44.431278 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gc9gd"] Oct 31 00:10:44.433016 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:44.432988 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3071e11c_bd77_4a11_bccf_f7635ec38ed9.slice/crio-ab76f5bde4702146f43bcf463bd1f6cb82dbc8cbe56eedcef925a8175962a2ac WatchSource:0}: Error finding container ab76f5bde4702146f43bcf463bd1f6cb82dbc8cbe56eedcef925a8175962a2ac: Status 404 returned error can't find the container with id ab76f5bde4702146f43bcf463bd1f6cb82dbc8cbe56eedcef925a8175962a2ac Oct 31 00:10:45.288828 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:45.288794 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" event={"ID":"3071e11c-bd77-4a11-bccf-f7635ec38ed9","Type":"ContainerStarted","Data":"ab76f5bde4702146f43bcf463bd1f6cb82dbc8cbe56eedcef925a8175962a2ac"} Oct 31 00:10:47.301421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:47.301372 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" event={"ID":"3071e11c-bd77-4a11-bccf-f7635ec38ed9","Type":"ContainerStarted","Data":"64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d"} Oct 31 00:10:47.301832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:47.301440 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:47.419361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:47.419306 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" podStartSLOduration=2.481659575 podStartE2EDuration="4.419291804s" totalImagesPullingTime="1.937632229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:47.417405098 +0000 UTC m=+479.892763522" watchObservedRunningTime="2026-10-31 00:10:47.419291804 +0000 UTC m=+479.894650227" Oct 31 00:10:48.197379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.197347 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9"] Oct 31 00:10:48.200334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.200312 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.243834 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.243804 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:10:48.243986 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.243917 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b5nfc" Oct 31 00:10:48.255795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.255765 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 00:10:48.255961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.255792 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 00:10:48.258187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.258165 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9"] Oct 31 00:10:48.325448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.325422 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgd76\" (UniqueName: \"kubernetes.io/projected/64c1ce0f-d843-4306-8d6f-8c37a72ec455-kube-api-access-kgd76\") pod \"kuadrant-console-plugin-66c4fb575d-dnvj9\" (UID: \"64c1ce0f-d843-4306-8d6f-8c37a72ec455\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.325898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.325569 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/64c1ce0f-d843-4306-8d6f-8c37a72ec455-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dnvj9\" (UID: \"64c1ce0f-d843-4306-8d6f-8c37a72ec455\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.325898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.325650 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/64c1ce0f-d843-4306-8d6f-8c37a72ec455-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dnvj9\" (UID: \"64c1ce0f-d843-4306-8d6f-8c37a72ec455\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.426566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.426532 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/64c1ce0f-d843-4306-8d6f-8c37a72ec455-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dnvj9\" (UID: \"64c1ce0f-d843-4306-8d6f-8c37a72ec455\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.426566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.426569 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/64c1ce0f-d843-4306-8d6f-8c37a72ec455-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dnvj9\" (UID: \"64c1ce0f-d843-4306-8d6f-8c37a72ec455\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.426771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.426652 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgd76\" (UniqueName: \"kubernetes.io/projected/64c1ce0f-d843-4306-8d6f-8c37a72ec455-kube-api-access-kgd76\") pod \"kuadrant-console-plugin-66c4fb575d-dnvj9\" (UID: \"64c1ce0f-d843-4306-8d6f-8c37a72ec455\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.427226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.427203 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/64c1ce0f-d843-4306-8d6f-8c37a72ec455-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dnvj9\" (UID: \"64c1ce0f-d843-4306-8d6f-8c37a72ec455\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.429180 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.429159 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/64c1ce0f-d843-4306-8d6f-8c37a72ec455-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dnvj9\" (UID: \"64c1ce0f-d843-4306-8d6f-8c37a72ec455\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.488271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.488198 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgd76\" (UniqueName: \"kubernetes.io/projected/64c1ce0f-d843-4306-8d6f-8c37a72ec455-kube-api-access-kgd76\") pod \"kuadrant-console-plugin-66c4fb575d-dnvj9\" (UID: \"64c1ce0f-d843-4306-8d6f-8c37a72ec455\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.511031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.511003 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" Oct 31 00:10:48.883527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:48.883505 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9"] Oct 31 00:10:48.885725 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:48.885698 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64c1ce0f_d843_4306_8d6f_8c37a72ec455.slice/crio-13e00a9c9e964300921a63b7c5176854e485b427d7396ee7eca16a8b88460ccb WatchSource:0}: Error finding container 13e00a9c9e964300921a63b7c5176854e485b427d7396ee7eca16a8b88460ccb: Status 404 returned error can't find the container with id 13e00a9c9e964300921a63b7c5176854e485b427d7396ee7eca16a8b88460ccb Oct 31 00:10:49.312859 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:49.312779 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" event={"ID":"64c1ce0f-d843-4306-8d6f-8c37a72ec455","Type":"ContainerStarted","Data":"13e00a9c9e964300921a63b7c5176854e485b427d7396ee7eca16a8b88460ccb"} Oct 31 00:10:53.279476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:53.279442 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zsk4m" Oct 31 00:10:55.350449 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:55.350409 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" event={"ID":"64c1ce0f-d843-4306-8d6f-8c37a72ec455","Type":"ContainerStarted","Data":"c76ac1af8c50691e8592184e6dd7001326be55c856018a9f8b1d93142e4f8f03"} Oct 31 00:10:55.467557 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:55.467496 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dnvj9" podStartSLOduration=1.607207574 podStartE2EDuration="7.467482257s" totalImagesPullingTime="5.860274683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:55.465453408 +0000 UTC m=+487.940811833" watchObservedRunningTime="2026-10-31 00:10:55.467482257 +0000 UTC m=+487.942840680" Oct 31 00:10:58.308155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.308076 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:10:58.625616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.625560 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm"] Oct 31 00:10:58.628310 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.628287 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.680866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.680839 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 00:10:58.680866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.680839 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-6dxfl\"" Oct 31 00:10:58.681025 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.680846 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 00:10:58.681025 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.680875 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 00:10:58.732384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.732354 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm"] Oct 31 00:10:58.819229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.819203 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45867a4d-8019-4dc8-adbc-53ee3c27b50d-metrics-certs\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.819403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.819234 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45867a4d-8019-4dc8-adbc-53ee3c27b50d-manager-config\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.819403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.819334 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45867a4d-8019-4dc8-adbc-53ee3c27b50d-cert\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.819403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.819393 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxmd\" (UniqueName: \"kubernetes.io/projected/45867a4d-8019-4dc8-adbc-53ee3c27b50d-kube-api-access-hjxmd\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.919846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.919763 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxmd\" (UniqueName: \"kubernetes.io/projected/45867a4d-8019-4dc8-adbc-53ee3c27b50d-kube-api-access-hjxmd\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.919846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.919816 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45867a4d-8019-4dc8-adbc-53ee3c27b50d-metrics-certs\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.920067 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.920053 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45867a4d-8019-4dc8-adbc-53ee3c27b50d-manager-config\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.920152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.920137 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45867a4d-8019-4dc8-adbc-53ee3c27b50d-cert\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.920721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.920699 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45867a4d-8019-4dc8-adbc-53ee3c27b50d-manager-config\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.922612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.922573 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45867a4d-8019-4dc8-adbc-53ee3c27b50d-metrics-certs\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.922707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.922574 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45867a4d-8019-4dc8-adbc-53ee3c27b50d-cert\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:58.980821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:58.980790 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxmd\" (UniqueName: \"kubernetes.io/projected/45867a4d-8019-4dc8-adbc-53ee3c27b50d-kube-api-access-hjxmd\") pod \"jobset-controller-manager-7b9d45fccb-s2gpm\" (UID: \"45867a4d-8019-4dc8-adbc-53ee3c27b50d\") " pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:59.238530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:59.238447 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:10:59.615029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:10:59.615001 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm"] Oct 31 00:10:59.616332 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:10:59.616305 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45867a4d_8019_4dc8_adbc_53ee3c27b50d.slice/crio-cb55d0220006e308f182b0cc5f5d47e41fada12888af8805f5628ab28e0a87ad WatchSource:0}: Error finding container cb55d0220006e308f182b0cc5f5d47e41fada12888af8805f5628ab28e0a87ad: Status 404 returned error can't find the container with id cb55d0220006e308f182b0cc5f5d47e41fada12888af8805f5628ab28e0a87ad Oct 31 00:11:00.378198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:00.378164 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" event={"ID":"45867a4d-8019-4dc8-adbc-53ee3c27b50d","Type":"ContainerStarted","Data":"cb55d0220006e308f182b0cc5f5d47e41fada12888af8805f5628ab28e0a87ad"} Oct 31 00:11:02.731498 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:11:02.731460 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:11:03.117890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:03.117851 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gc9gd"] Oct 31 00:11:03.118112 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:03.118087 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" podUID="3071e11c-bd77-4a11-bccf-f7635ec38ed9" containerName="limitador" containerID="cri-o://64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d" gracePeriod=30 Oct 31 00:11:03.700533 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:03.700508 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:11:03.755459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:03.755427 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3071e11c-bd77-4a11-bccf-f7635ec38ed9-kube-api-access-p2cmp\" (UniqueName: \"kubernetes.io/projected/3071e11c-bd77-4a11-bccf-f7635ec38ed9-kube-api-access-p2cmp\") pod \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\" (UID: \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\") " Oct 31 00:11:03.755889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:03.755515 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3071e11c-bd77-4a11-bccf-f7635ec38ed9-config-file\" (UniqueName: \"kubernetes.io/configmap/3071e11c-bd77-4a11-bccf-f7635ec38ed9-config-file\") pod \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\" (UID: \"3071e11c-bd77-4a11-bccf-f7635ec38ed9\") " Oct 31 00:11:03.756617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:03.756561 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3071e11c-bd77-4a11-bccf-f7635ec38ed9-config-file" pod "3071e11c-bd77-4a11-bccf-f7635ec38ed9" (UID: "3071e11c-bd77-4a11-bccf-f7635ec38ed9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:11:03.758620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:03.758575 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3071e11c-bd77-4a11-bccf-f7635ec38ed9-kube-api-access-p2cmp" pod "3071e11c-bd77-4a11-bccf-f7635ec38ed9" (UID: "3071e11c-bd77-4a11-bccf-f7635ec38ed9"). InnerVolumeSpecName "kube-api-access-p2cmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:11:03.856453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:03.856413 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2cmp\" (UniqueName: \"kubernetes.io/projected/3071e11c-bd77-4a11-bccf-f7635ec38ed9-kube-api-access-p2cmp\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:11:03.856453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:03.856458 4969 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3071e11c-bd77-4a11-bccf-f7635ec38ed9-config-file\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:11:04.402965 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.402929 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" event={"ID":"45867a4d-8019-4dc8-adbc-53ee3c27b50d","Type":"ContainerStarted","Data":"df81bfdff1a92e392f4027b22984bdcc0d44a2b788967ef2e8a961f8e77d2431"} Oct 31 00:11:04.403163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.403030 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:11:04.404383 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.404353 4969 generic.go:356] "Generic (PLEG): container finished" podID="3071e11c-bd77-4a11-bccf-f7635ec38ed9" containerID="64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d" exitCode=0 Oct 31 00:11:04.404511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.404413 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" Oct 31 00:11:04.404511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.404410 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" event={"ID":"3071e11c-bd77-4a11-bccf-f7635ec38ed9","Type":"ContainerDied","Data":"64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d"} Oct 31 00:11:04.404602 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.404535 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-gc9gd" event={"ID":"3071e11c-bd77-4a11-bccf-f7635ec38ed9","Type":"ContainerDied","Data":"ab76f5bde4702146f43bcf463bd1f6cb82dbc8cbe56eedcef925a8175962a2ac"} Oct 31 00:11:04.404602 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.404559 4969 scope.go:122] "RemoveContainer" containerID="64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d" Oct 31 00:11:04.418020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.417996 4969 scope.go:122] "RemoveContainer" containerID="64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d" Oct 31 00:11:04.418313 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:11:04.418293 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d\": container with ID starting with 64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d not found: ID does not exist" containerID="64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d" Oct 31 00:11:04.418381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.418327 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d"} err="failed to get container status \"64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d\": rpc error: code = NotFound desc = could not find container \"64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d\": container with ID starting with 64993bebac8e9647c695aa6976f53e045d34d70c4fd968c5b93cb67d3b5ec94d not found: ID does not exist" Oct 31 00:11:04.519301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.519238 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" podStartSLOduration=2.110429043 podStartE2EDuration="6.519217479s" totalImagesPullingTime="4.408788436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:04.518554791 +0000 UTC m=+496.993913215" watchObservedRunningTime="2026-10-31 00:11:04.519217479 +0000 UTC m=+496.994575904" Oct 31 00:11:04.629841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.629804 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gc9gd"] Oct 31 00:11:04.635645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:04.635613 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-gc9gd"] Oct 31 00:11:05.732961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:05.732918 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3071e11c-bd77-4a11-bccf-f7635ec38ed9" path="/var/lib/kubelet/pods/3071e11c-bd77-4a11-bccf-f7635ec38ed9/volumes" Oct 31 00:11:06.043816 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:06.043734 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf"] Oct 31 00:11:06.043966 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:06.043924 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" podUID="dec2cccd-e151-48f1-b2c5-ec2960453d80" containerName="manager" containerID="cri-o://435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a" gracePeriod=10 Oct 31 00:11:07.129291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.129263 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:11:07.182505 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.182468 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec2cccd-e151-48f1-b2c5-ec2960453d80-kube-api-access-xlr9q\" (UniqueName: \"kubernetes.io/projected/dec2cccd-e151-48f1-b2c5-ec2960453d80-kube-api-access-xlr9q\") pod \"dec2cccd-e151-48f1-b2c5-ec2960453d80\" (UID: \"dec2cccd-e151-48f1-b2c5-ec2960453d80\") " Oct 31 00:11:07.182709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.182542 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec2cccd-e151-48f1-b2c5-ec2960453d80-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec2cccd-e151-48f1-b2c5-ec2960453d80-extensions-socket-volume\") pod \"dec2cccd-e151-48f1-b2c5-ec2960453d80\" (UID: \"dec2cccd-e151-48f1-b2c5-ec2960453d80\") " Oct 31 00:11:07.182923 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.182900 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec2cccd-e151-48f1-b2c5-ec2960453d80-extensions-socket-volume" pod "dec2cccd-e151-48f1-b2c5-ec2960453d80" (UID: "dec2cccd-e151-48f1-b2c5-ec2960453d80"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:11:07.184847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.184821 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec2cccd-e151-48f1-b2c5-ec2960453d80-kube-api-access-xlr9q" pod "dec2cccd-e151-48f1-b2c5-ec2960453d80" (UID: "dec2cccd-e151-48f1-b2c5-ec2960453d80"). InnerVolumeSpecName "kube-api-access-xlr9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:11:07.284156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.284085 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlr9q\" (UniqueName: \"kubernetes.io/projected/dec2cccd-e151-48f1-b2c5-ec2960453d80-kube-api-access-xlr9q\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:11:07.284156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.284109 4969 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dec2cccd-e151-48f1-b2c5-ec2960453d80-extensions-socket-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:11:07.423970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.423935 4969 generic.go:356] "Generic (PLEG): container finished" podID="dec2cccd-e151-48f1-b2c5-ec2960453d80" containerID="435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a" exitCode=0 Oct 31 00:11:07.424128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.423996 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" Oct 31 00:11:07.424128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.424015 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" event={"ID":"dec2cccd-e151-48f1-b2c5-ec2960453d80","Type":"ContainerDied","Data":"435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a"} Oct 31 00:11:07.424128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.424073 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf" event={"ID":"dec2cccd-e151-48f1-b2c5-ec2960453d80","Type":"ContainerDied","Data":"9d7b2a4c4d44db689d70ce3a92eef89e7a320f8bd4d28833b310d326ab8633f1"} Oct 31 00:11:07.424128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.424086 4969 scope.go:122] "RemoveContainer" containerID="435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a" Oct 31 00:11:07.433343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.433315 4969 scope.go:122] "RemoveContainer" containerID="435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a" Oct 31 00:11:07.433635 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:11:07.433613 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a\": container with ID starting with 435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a not found: ID does not exist" containerID="435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a" Oct 31 00:11:07.433692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.433647 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a"} err="failed to get container status \"435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a\": rpc error: code = NotFound desc = could not find container \"435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a\": container with ID starting with 435ed2e36242661c74a2be43403097febb257586d2b229022a32fd469beb857a not found: ID does not exist" Oct 31 00:11:07.588906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.588831 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf"] Oct 31 00:11:07.595037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.595014 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-csnwf"] Oct 31 00:11:07.732486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:07.732455 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec2cccd-e151-48f1-b2c5-ec2960453d80" path="/var/lib/kubelet/pods/dec2cccd-e151-48f1-b2c5-ec2960453d80/volumes" Oct 31 00:11:10.319115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.319079 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r"] Oct 31 00:11:10.319566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.319416 4969 cpu_manager.go:414] "containerMap: removing container" podUID="dec2cccd-e151-48f1-b2c5-ec2960453d80" containerName="manager" Oct 31 00:11:10.319566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.319427 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec2cccd-e151-48f1-b2c5-ec2960453d80" containerName="manager" Oct 31 00:11:10.319566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.319436 4969 cpu_manager.go:414] "containerMap: removing container" podUID="3071e11c-bd77-4a11-bccf-f7635ec38ed9" containerName="limitador" Oct 31 00:11:10.319566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.319441 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3071e11c-bd77-4a11-bccf-f7635ec38ed9" containerName="limitador" Oct 31 00:11:10.319566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.319514 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec2cccd-e151-48f1-b2c5-ec2960453d80" containerName="manager" Oct 31 00:11:10.319566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.319520 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="3071e11c-bd77-4a11-bccf-f7635ec38ed9" containerName="limitador" Oct 31 00:11:10.322071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.322056 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:10.380902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.380873 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:11:10.381044 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.380873 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 00:11:10.383726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.383709 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:11:10.408932 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.408907 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b522953e-42b0-4a9e-82db-145e50b8abe3-data\") pod \"postgres-7cdbf6dc44-4ld6r\" (UID: \"b522953e-42b0-4a9e-82db-145e50b8abe3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:10.409066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.408945 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmqd\" (UniqueName: \"kubernetes.io/projected/b522953e-42b0-4a9e-82db-145e50b8abe3-kube-api-access-5tmqd\") pod \"postgres-7cdbf6dc44-4ld6r\" (UID: \"b522953e-42b0-4a9e-82db-145e50b8abe3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:10.430785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.430766 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-89sjz\"" Oct 31 00:11:10.432771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.432752 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r"] Oct 31 00:11:10.510035 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.510000 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b522953e-42b0-4a9e-82db-145e50b8abe3-data\") pod \"postgres-7cdbf6dc44-4ld6r\" (UID: \"b522953e-42b0-4a9e-82db-145e50b8abe3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:10.510201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.510038 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmqd\" (UniqueName: \"kubernetes.io/projected/b522953e-42b0-4a9e-82db-145e50b8abe3-kube-api-access-5tmqd\") pod \"postgres-7cdbf6dc44-4ld6r\" (UID: \"b522953e-42b0-4a9e-82db-145e50b8abe3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:10.510381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.510363 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b522953e-42b0-4a9e-82db-145e50b8abe3-data\") pod \"postgres-7cdbf6dc44-4ld6r\" (UID: \"b522953e-42b0-4a9e-82db-145e50b8abe3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:10.571185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.571110 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmqd\" (UniqueName: \"kubernetes.io/projected/b522953e-42b0-4a9e-82db-145e50b8abe3-kube-api-access-5tmqd\") pod \"postgres-7cdbf6dc44-4ld6r\" (UID: \"b522953e-42b0-4a9e-82db-145e50b8abe3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:10.633234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:10.633202 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:11.001778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:11.001757 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r"] Oct 31 00:11:11.004091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:11:11.004065 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb522953e_42b0_4a9e_82db_145e50b8abe3.slice/crio-49acce4bb4f6d7f2f7c1e25f998ba65fef37f6e74e77c24a596fbd70542c369c WatchSource:0}: Error finding container 49acce4bb4f6d7f2f7c1e25f998ba65fef37f6e74e77c24a596fbd70542c369c: Status 404 returned error can't find the container with id 49acce4bb4f6d7f2f7c1e25f998ba65fef37f6e74e77c24a596fbd70542c369c Oct 31 00:11:11.446445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:11.446413 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" event={"ID":"b522953e-42b0-4a9e-82db-145e50b8abe3","Type":"ContainerStarted","Data":"49acce4bb4f6d7f2f7c1e25f998ba65fef37f6e74e77c24a596fbd70542c369c"} Oct 31 00:11:15.417150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:15.417116 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7b9d45fccb-s2gpm" Oct 31 00:11:17.486716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:17.486669 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" event={"ID":"b522953e-42b0-4a9e-82db-145e50b8abe3","Type":"ContainerStarted","Data":"907b11ce0fa680a5f91d8e33e885c23d907d09640ecbc2aa6ccda91f43c46842"} Oct 31 00:11:17.487226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:17.486867 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:17.601105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:17.600978 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" podStartSLOduration=1.32887643 podStartE2EDuration="7.600957481s" totalImagesPullingTime="6.272081051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:17.600543726 +0000 UTC m=+510.075902149" watchObservedRunningTime="2026-10-31 00:11:17.600957481 +0000 UTC m=+510.076315895" Oct 31 00:11:23.522821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:23.522786 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4ld6r" Oct 31 00:11:32.729699 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:11:32.729664 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:11:46.741226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.741192 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj"] Oct 31 00:11:46.747725 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.747697 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:46.803017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.802991 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj"] Oct 31 00:11:46.850087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.850047 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 00:11:46.850279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.850094 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 00:11:46.850279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.850130 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 00:11:46.850279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.850176 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 00:11:46.850279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.850207 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 00:11:46.850279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.850251 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-cqkxg\"" Oct 31 00:11:46.927149 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.927118 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4ce42331-4673-4ade-add7-cc6a77a308ec-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:46.927301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.927165 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ce42331-4673-4ade-add7-cc6a77a308ec-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:46.927301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.927231 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnz4z\" (UniqueName: \"kubernetes.io/projected/4ce42331-4673-4ade-add7-cc6a77a308ec-kube-api-access-gnz4z\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:46.927301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:46.927274 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce42331-4673-4ade-add7-cc6a77a308ec-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.028640 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.028518 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz4z\" (UniqueName: \"kubernetes.io/projected/4ce42331-4673-4ade-add7-cc6a77a308ec-kube-api-access-gnz4z\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.028640 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.028578 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce42331-4673-4ade-add7-cc6a77a308ec-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.028889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.028737 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4ce42331-4673-4ade-add7-cc6a77a308ec-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.028889 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:11:47.028848 4969 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 31 00:11:47.029003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.028887 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ce42331-4673-4ade-add7-cc6a77a308ec-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.029003 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:11:47.028929 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ce42331-4673-4ade-add7-cc6a77a308ec-proxy-source podName:4ce42331-4673-4ade-add7-cc6a77a308ec nodeName:}" failed. No retries permitted until 2026-10-31 00:11:47.528907828 +0000 UTC m=+540.004266246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/4ce42331-4673-4ade-add7-cc6a77a308ec-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-svrwj" (UID: "4ce42331-4673-4ade-add7-cc6a77a308ec") : configmap "usage-tenancy-proxy-source" not found Oct 31 00:11:47.029217 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.029199 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce42331-4673-4ade-add7-cc6a77a308ec-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.031468 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.031450 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ce42331-4673-4ade-add7-cc6a77a308ec-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.092002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.091976 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnz4z\" (UniqueName: \"kubernetes.io/projected/4ce42331-4673-4ade-add7-cc6a77a308ec-kube-api-access-gnz4z\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.534551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.534515 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4ce42331-4673-4ade-add7-cc6a77a308ec-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.535317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.535298 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4ce42331-4673-4ade-add7-cc6a77a308ec-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-svrwj\" (UID: \"4ce42331-4673-4ade-add7-cc6a77a308ec\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:47.711111 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.711080 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-cqkxg\"" Oct 31 00:11:47.720204 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:47.720175 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:11:48.097659 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:48.097635 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj"] Oct 31 00:11:48.099652 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:11:48.099621 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce42331_4673_4ade_add7_cc6a77a308ec.slice/crio-a683375282813b04610002e5a8525951d292fe00b5ca8ee3812c6ceb1a4e07f1 WatchSource:0}: Error finding container a683375282813b04610002e5a8525951d292fe00b5ca8ee3812c6ceb1a4e07f1: Status 404 returned error can't find the container with id a683375282813b04610002e5a8525951d292fe00b5ca8ee3812c6ceb1a4e07f1 Oct 31 00:11:48.656825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:11:48.656773 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" event={"ID":"4ce42331-4673-4ade-add7-cc6a77a308ec","Type":"ContainerStarted","Data":"a683375282813b04610002e5a8525951d292fe00b5ca8ee3812c6ceb1a4e07f1"} Oct 31 00:12:01.741505 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:01.741469 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" event={"ID":"4ce42331-4673-4ade-add7-cc6a77a308ec","Type":"ContainerStarted","Data":"fcefa859ccc93b849c829020b40ad3170c9522cb86abab11a8a9f96cc88b103b"} Oct 31 00:12:01.741941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:01.741524 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:12:01.859038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:01.858982 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" podStartSLOduration=2.6677946 podStartE2EDuration="15.858970294s" totalImagesPullingTime="13.191175694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:01.858859397 +0000 UTC m=+554.334217821" watchObservedRunningTime="2026-10-31 00:12:01.858970294 +0000 UTC m=+554.334328717" Oct 31 00:12:02.726813 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:12:02.726776 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:12:12.756572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:12.756542 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-svrwj" Oct 31 00:12:12.925526 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:12.925495 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jgwjj"] Oct 31 00:12:12.929166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:12.929144 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:12.964608 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:12.964560 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jgwjj"] Oct 31 00:12:12.984422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:12.984383 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fwh2t\"" Oct 31 00:12:12.984559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:12.984489 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 00:12:13.059983 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:13.059896 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9cbf01-d844-41ce-8778-56c3d92f9862-tls-certs\") pod \"model-serving-api-75bd89fdfc-jgwjj\" (UID: \"fb9cbf01-d844-41ce-8778-56c3d92f9862\") " pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:13.059983 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:13.059942 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwzb\" (UniqueName: \"kubernetes.io/projected/fb9cbf01-d844-41ce-8778-56c3d92f9862-kube-api-access-whwzb\") pod \"model-serving-api-75bd89fdfc-jgwjj\" (UID: \"fb9cbf01-d844-41ce-8778-56c3d92f9862\") " pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:13.160497 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:13.160464 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9cbf01-d844-41ce-8778-56c3d92f9862-tls-certs\") pod \"model-serving-api-75bd89fdfc-jgwjj\" (UID: \"fb9cbf01-d844-41ce-8778-56c3d92f9862\") " pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:13.160497 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:13.160496 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwzb\" (UniqueName: \"kubernetes.io/projected/fb9cbf01-d844-41ce-8778-56c3d92f9862-kube-api-access-whwzb\") pod \"model-serving-api-75bd89fdfc-jgwjj\" (UID: \"fb9cbf01-d844-41ce-8778-56c3d92f9862\") " pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:13.163194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:13.163162 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb9cbf01-d844-41ce-8778-56c3d92f9862-tls-certs\") pod \"model-serving-api-75bd89fdfc-jgwjj\" (UID: \"fb9cbf01-d844-41ce-8778-56c3d92f9862\") " pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:13.221100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:13.221072 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwzb\" (UniqueName: \"kubernetes.io/projected/fb9cbf01-d844-41ce-8778-56c3d92f9862-kube-api-access-whwzb\") pod \"model-serving-api-75bd89fdfc-jgwjj\" (UID: \"fb9cbf01-d844-41ce-8778-56c3d92f9862\") " pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:13.242014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:13.241991 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:13.613961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:13.613933 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jgwjj"] Oct 31 00:12:13.615662 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:12:13.615632 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9cbf01_d844_41ce_8778_56c3d92f9862.slice/crio-9511aae290a8493d16630cac8e230e35d4f17bb91bd4f9b38452a41574246e10 WatchSource:0}: Error finding container 9511aae290a8493d16630cac8e230e35d4f17bb91bd4f9b38452a41574246e10: Status 404 returned error can't find the container with id 9511aae290a8493d16630cac8e230e35d4f17bb91bd4f9b38452a41574246e10 Oct 31 00:12:13.810788 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:13.810752 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" event={"ID":"fb9cbf01-d844-41ce-8778-56c3d92f9862","Type":"ContainerStarted","Data":"9511aae290a8493d16630cac8e230e35d4f17bb91bd4f9b38452a41574246e10"} Oct 31 00:12:15.828558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:15.828510 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" event={"ID":"fb9cbf01-d844-41ce-8778-56c3d92f9862","Type":"ContainerStarted","Data":"febbfd3ace7da2c62cf14b389c406e19d674888ff17059ad8302f0bd26483d4d"} Oct 31 00:12:15.829207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:15.829184 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:15.945833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:15.945722 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" podStartSLOduration=1.8880127309999999 podStartE2EDuration="3.945707735s" totalImagesPullingTime="2.057695004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:15.945102559 +0000 UTC m=+568.420460984" watchObservedRunningTime="2026-10-31 00:12:15.945707735 +0000 UTC m=+568.421066157" Oct 31 00:12:26.839946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:26.839918 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-jgwjj" Oct 31 00:12:32.726578 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:12:32.726468 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:12:32.942903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:32.942872 4969 generic.go:356] "Generic (PLEG): container finished" podID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerID="f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc" exitCode=0 Oct 31 00:12:32.943091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:32.942946 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" event={"ID":"2f6e4058-e548-4a56-9017-5b57dad353ce","Type":"ContainerDied","Data":"f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc"} Oct 31 00:12:32.943345 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:32.943331 4969 scope.go:122] "RemoveContainer" containerID="f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc" Oct 31 00:12:32.944389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:32.944356 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:32.944473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:32.944425 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:32.944473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:32.944457 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:33.950344 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:33.950309 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" event={"ID":"2f6e4058-e548-4a56-9017-5b57dad353ce","Type":"ContainerStarted","Data":"b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4"} Oct 31 00:12:34.788944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:34.788819 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:12:34.790652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:34.790623 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-thkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:12:34.790885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:34.790860 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 00:12:35.793618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:35.792972 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-thkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:12:35.793618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:35.793037 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 00:12:36.788048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:36.788009 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-thkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:12:36.788354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:36.788316 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 00:12:37.797619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:37.783988 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-thkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:12:37.797619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:37.784087 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 00:12:38.566309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:38.566264 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:12:38.792621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:38.783982 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-thkzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 31 00:12:38.792621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:38.784062 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 31 00:12:39.787423 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:39.787395 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:12:39.788440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:39.788421 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:12:47.759636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:47.759609 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:12:47.761186 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:47.761160 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:12:47.774548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:47.774528 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:12:47.776518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:12:47.776489 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:13:02.726868 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:13:02.726829 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:13:32.726188 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:13:32.726147 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:14:02.727160 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:14:02.727070 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:14:32.725586 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:14:32.725547 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:14:51.160652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.160620 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d"] Oct 31 00:14:51.162857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.162840 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.245282 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.245247 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.245471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.245307 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.245471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.245395 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5337c97b-b317-4aaf-a6fd-a94fa2466175-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.245471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.245442 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.245583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.245523 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.245583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.245560 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8vqd\" (UniqueName: \"kubernetes.io/projected/5337c97b-b317-4aaf-a6fd-a94fa2466175-kube-api-access-p8vqd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.265154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.265122 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 00:14:51.265290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.265176 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 00:14:51.265290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.265211 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 00:14:51.265366 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.265346 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4mp6x\"" Oct 31 00:14:51.267520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.267504 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d"] Oct 31 00:14:51.346895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.346866 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.347055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.346901 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5337c97b-b317-4aaf-a6fd-a94fa2466175-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.347055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.346925 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.347055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.346955 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.347055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.346991 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8vqd\" (UniqueName: \"kubernetes.io/projected/5337c97b-b317-4aaf-a6fd-a94fa2466175-kube-api-access-p8vqd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.347270 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.347080 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.347353 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.347331 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.347413 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.347328 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.347454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.347418 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.349433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.349412 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5337c97b-b317-4aaf-a6fd-a94fa2466175-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.349700 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.349680 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5337c97b-b317-4aaf-a6fd-a94fa2466175-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.408247 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.408220 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8vqd\" (UniqueName: \"kubernetes.io/projected/5337c97b-b317-4aaf-a6fd-a94fa2466175-kube-api-access-p8vqd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d\" (UID: \"5337c97b-b317-4aaf-a6fd-a94fa2466175\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.474087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.474020 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:14:51.848610 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.848574 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d"] Oct 31 00:14:51.850214 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:14:51.850190 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5337c97b_b317_4aaf_a6fd_a94fa2466175.slice/crio-f119166ddf4051bad08c06afeacd9e94e2ca43d17e1522b9f89673e7e1fb1a43 WatchSource:0}: Error finding container f119166ddf4051bad08c06afeacd9e94e2ca43d17e1522b9f89673e7e1fb1a43: Status 404 returned error can't find the container with id f119166ddf4051bad08c06afeacd9e94e2ca43d17e1522b9f89673e7e1fb1a43 Oct 31 00:14:51.851483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:51.851463 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:14:52.710912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:52.710872 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" event={"ID":"5337c97b-b317-4aaf-a6fd-a94fa2466175","Type":"ContainerStarted","Data":"f119166ddf4051bad08c06afeacd9e94e2ca43d17e1522b9f89673e7e1fb1a43"} Oct 31 00:14:54.726132 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:54.726096 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" event={"ID":"5337c97b-b317-4aaf-a6fd-a94fa2466175","Type":"ContainerStarted","Data":"cffd3eaee0db0ccb6f0c380d95f342726f5704cd291a17fcab3cf8c2eb085751"} Oct 31 00:14:54.846169 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:54.846124 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" podStartSLOduration=1.9609894350000001 podStartE2EDuration="3.846110465s" totalImagesPullingTime="1.88512103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:54.843766212 +0000 UTC m=+727.319124632" watchObservedRunningTime="2026-10-31 00:14:54.846110465 +0000 UTC m=+727.321468887" Oct 31 00:14:55.474203 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:14:55.474169 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:15:00.167852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.167812 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv"] Oct 31 00:15:00.171647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.171626 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:00.229894 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.229861 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv"] Oct 31 00:15:00.275988 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.275954 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:15:00.330740 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.330701 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hdr\" (UniqueName: \"kubernetes.io/projected/129ee960-8884-4992-8468-053b10cbe557-kube-api-access-b4hdr\") pod \"maas-api-key-cleanup-29890095-27pmv\" (UID: \"129ee960-8884-4992-8468-053b10cbe557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:00.330905 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.330827 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee960-8884-4992-8468-053b10cbe557-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-27pmv\" (UID: \"129ee960-8884-4992-8468-053b10cbe557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:00.431538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.431441 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee960-8884-4992-8468-053b10cbe557-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-27pmv\" (UID: \"129ee960-8884-4992-8468-053b10cbe557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:00.431538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.431523 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hdr\" (UniqueName: \"kubernetes.io/projected/129ee960-8884-4992-8468-053b10cbe557-kube-api-access-b4hdr\") pod \"maas-api-key-cleanup-29890095-27pmv\" (UID: \"129ee960-8884-4992-8468-053b10cbe557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:00.432154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.432128 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee960-8884-4992-8468-053b10cbe557-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-27pmv\" (UID: \"129ee960-8884-4992-8468-053b10cbe557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:00.494156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.494125 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hdr\" (UniqueName: \"kubernetes.io/projected/129ee960-8884-4992-8468-053b10cbe557-kube-api-access-b4hdr\") pod \"maas-api-key-cleanup-29890095-27pmv\" (UID: \"129ee960-8884-4992-8468-053b10cbe557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:00.783779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:00.783679 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:01.164311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.164270 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv"] Oct 31 00:15:01.168420 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:15:01.168388 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129ee960_8884_4992_8468_053b10cbe557.slice/crio-5f95aeb4a70fd92c41aa9bcaaf681bb664a5eabe2d5fb1d6539cf85a120ecb8c WatchSource:0}: Error finding container 5f95aeb4a70fd92c41aa9bcaaf681bb664a5eabe2d5fb1d6539cf85a120ecb8c: Status 404 returned error can't find the container with id 5f95aeb4a70fd92c41aa9bcaaf681bb664a5eabe2d5fb1d6539cf85a120ecb8c Oct 31 00:15:01.475269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.475185 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:15:01.488071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.488044 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:15:01.757869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.757797 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x"] Oct 31 00:15:01.761263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.761246 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.765239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.765209 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" event={"ID":"129ee960-8884-4992-8468-053b10cbe557","Type":"ContainerStarted","Data":"5f95aeb4a70fd92c41aa9bcaaf681bb664a5eabe2d5fb1d6539cf85a120ecb8c"} Oct 31 00:15:01.777070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.777046 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d" Oct 31 00:15:01.814023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.813997 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 00:15:01.820116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.820088 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x"] Oct 31 00:15:01.843855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.843821 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.843855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.843855 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.844035 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.843881 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.844035 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.843989 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.844129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.844113 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.844178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.844149 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q84dq\" (UniqueName: \"kubernetes.io/projected/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-kube-api-access-q84dq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.945437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.945397 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.945437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.945437 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q84dq\" (UniqueName: \"kubernetes.io/projected/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-kube-api-access-q84dq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.945735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.945482 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.945735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.945506 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.945735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.945539 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.945735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.945575 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.945976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.945911 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.946128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.946105 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.946178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.946162 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.948108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.948079 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:01.948259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:01.948242 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:02.007109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:02.007072 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q84dq\" (UniqueName: \"kubernetes.io/projected/2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52-kube-api-access-q84dq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5qn4x\" (UID: \"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:02.075177 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:02.075087 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:02.457499 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:02.457475 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x"] Oct 31 00:15:02.459126 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:15:02.459098 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd6f9fd_aafe_4a92_ac8c_53a0944d2b52.slice/crio-3878e4e7e7e5216a8b1945e53f5a5f58c2651e45739c1ea8b2f06d1c310ef81e WatchSource:0}: Error finding container 3878e4e7e7e5216a8b1945e53f5a5f58c2651e45739c1ea8b2f06d1c310ef81e: Status 404 returned error can't find the container with id 3878e4e7e7e5216a8b1945e53f5a5f58c2651e45739c1ea8b2f06d1c310ef81e Oct 31 00:15:02.726072 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:15:02.725982 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:15:02.772479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:02.772446 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" event={"ID":"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52","Type":"ContainerStarted","Data":"3878e4e7e7e5216a8b1945e53f5a5f58c2651e45739c1ea8b2f06d1c310ef81e"} Oct 31 00:15:03.781645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:03.781579 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" event={"ID":"2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52","Type":"ContainerStarted","Data":"87460dd0ddb164c223c4ff0883b3bb704442b3ff92890603aba1f547c54f42f5"} Oct 31 00:15:03.901885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:03.901823 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" podStartSLOduration=2.237268876 podStartE2EDuration="2.901806533s" totalImagesPullingTime="664.537657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:03.898241941 +0000 UTC m=+736.373600366" watchObservedRunningTime="2026-10-31 00:15:03.901806533 +0000 UTC m=+736.377164958" Oct 31 00:15:04.076049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:04.075968 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:05.795509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:05.795419 4969 generic.go:356] "Generic (PLEG): container finished" podID="129ee960-8884-4992-8468-053b10cbe557" containerID="5fa8552160c9405c2f9ca591e210a859d36c6eb664a4228110067d82cdeec9de" exitCode=60 Oct 31 00:15:05.796004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:05.795512 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" event={"ID":"129ee960-8884-4992-8468-053b10cbe557","Type":"ContainerDied","Data":"5fa8552160c9405c2f9ca591e210a859d36c6eb664a4228110067d82cdeec9de"} Oct 31 00:15:05.796004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:05.795717 4969 scope.go:122] "RemoveContainer" containerID="5fa8552160c9405c2f9ca591e210a859d36c6eb664a4228110067d82cdeec9de" Oct 31 00:15:06.810173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:06.810129 4969 generic.go:356] "Generic (PLEG): container finished" podID="129ee960-8884-4992-8468-053b10cbe557" containerID="3297bdb35e73766e364b93471943734aa029d8f533c751f0672c4b849f1c7b02" exitCode=60 Oct 31 00:15:06.810614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:06.810203 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" event={"ID":"129ee960-8884-4992-8468-053b10cbe557","Type":"ContainerDied","Data":"3297bdb35e73766e364b93471943734aa029d8f533c751f0672c4b849f1c7b02"} Oct 31 00:15:06.810614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:06.810286 4969 scope.go:122] "RemoveContainer" containerID="5fa8552160c9405c2f9ca591e210a859d36c6eb664a4228110067d82cdeec9de" Oct 31 00:15:06.810614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:06.810473 4969 scope.go:122] "RemoveContainer" containerID="3297bdb35e73766e364b93471943734aa029d8f533c751f0672c4b849f1c7b02" Oct 31 00:15:06.810748 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:15:06.810729 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-27pmv_odh-ai-gateway-infra(129ee960-8884-4992-8468-053b10cbe557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" podUID="129ee960-8884-4992-8468-053b10cbe557" Oct 31 00:15:07.817980 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:07.817952 4969 scope.go:122] "RemoveContainer" containerID="3297bdb35e73766e364b93471943734aa029d8f533c751f0672c4b849f1c7b02" Oct 31 00:15:07.818363 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:15:07.818181 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-27pmv_odh-ai-gateway-infra(129ee960-8884-4992-8468-053b10cbe557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" podUID="129ee960-8884-4992-8468-053b10cbe557" Oct 31 00:15:10.906337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:10.906298 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck"] Oct 31 00:15:10.911348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:10.911324 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:10.932057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:10.932027 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:10.932173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:10.932072 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:10.932173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:10.932163 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:10.932307 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:10.932284 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:10.932366 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:10.932323 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbtj6\" (UniqueName: \"kubernetes.io/projected/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-kube-api-access-mbtj6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:10.932416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:10.932359 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.013138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.013110 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 00:15:11.016111 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.016081 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck"] Oct 31 00:15:11.033692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.033655 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.033842 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.033738 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.033842 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.033771 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbtj6\" (UniqueName: \"kubernetes.io/projected/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-kube-api-access-mbtj6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.033842 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.033807 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.034029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.033898 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.034029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.033923 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.034142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.034067 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.034432 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.034409 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.034633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.034416 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.036465 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.036437 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.037251 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.037226 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.095270 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.095233 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbtj6\" (UniqueName: \"kubernetes.io/projected/783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d-kube-api-access-mbtj6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck\" (UID: \"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.228866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.228770 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:11.605117 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.605088 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck"] Oct 31 00:15:11.606323 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:15:11.606279 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783e7d8f_b5bd_4ef4_8dbd_d3f3309e9b3d.slice/crio-6a863351f8f06d680858b055142a1f2e11afe7bc0212aeef8fff7a4b95f844b6 WatchSource:0}: Error finding container 6a863351f8f06d680858b055142a1f2e11afe7bc0212aeef8fff7a4b95f844b6: Status 404 returned error can't find the container with id 6a863351f8f06d680858b055142a1f2e11afe7bc0212aeef8fff7a4b95f844b6 Oct 31 00:15:11.841585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:11.841553 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" event={"ID":"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d","Type":"ContainerStarted","Data":"6a863351f8f06d680858b055142a1f2e11afe7bc0212aeef8fff7a4b95f844b6"} Oct 31 00:15:12.075506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:12.075470 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:12.087666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:12.087647 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:12.848279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:12.848248 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" event={"ID":"783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d","Type":"ContainerStarted","Data":"0737df3129b559c0e00e592cfd327283952cae31e6e29d80388e240340822035"} Oct 31 00:15:12.859363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:12.859341 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5qn4x" Oct 31 00:15:12.967145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:12.967100 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" podStartSLOduration=2.37294686 podStartE2EDuration="2.967083482s" totalImagesPullingTime="594.136622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:12.966279882 +0000 UTC m=+745.441638306" watchObservedRunningTime="2026-10-31 00:15:12.967083482 +0000 UTC m=+745.442441905" Oct 31 00:15:13.229584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:13.229550 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:17.730980 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:17.730950 4969 scope.go:122] "RemoveContainer" containerID="3297bdb35e73766e364b93471943734aa029d8f533c751f0672c4b849f1c7b02" Oct 31 00:15:18.885396 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:18.885364 4969 generic.go:356] "Generic (PLEG): container finished" podID="129ee960-8884-4992-8468-053b10cbe557" containerID="635fef01cda42e148a4231224e6a3e1ea3e47f6b398d82d86fe1f0bb7b6dc649" exitCode=60 Oct 31 00:15:18.885875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:18.885437 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" event={"ID":"129ee960-8884-4992-8468-053b10cbe557","Type":"ContainerDied","Data":"635fef01cda42e148a4231224e6a3e1ea3e47f6b398d82d86fe1f0bb7b6dc649"} Oct 31 00:15:18.885875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:18.885509 4969 scope.go:122] "RemoveContainer" containerID="3297bdb35e73766e364b93471943734aa029d8f533c751f0672c4b849f1c7b02" Oct 31 00:15:18.885965 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:18.885888 4969 scope.go:122] "RemoveContainer" containerID="635fef01cda42e148a4231224e6a3e1ea3e47f6b398d82d86fe1f0bb7b6dc649" Oct 31 00:15:18.886160 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:15:18.886139 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-27pmv_odh-ai-gateway-infra(129ee960-8884-4992-8468-053b10cbe557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" podUID="129ee960-8884-4992-8468-053b10cbe557" Oct 31 00:15:20.025491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.025455 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv"] Oct 31 00:15:20.161584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.161555 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:20.216809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.216778 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129ee960-8884-4992-8468-053b10cbe557-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee960-8884-4992-8468-053b10cbe557-maas-api-service-ca\") pod \"129ee960-8884-4992-8468-053b10cbe557\" (UID: \"129ee960-8884-4992-8468-053b10cbe557\") " Oct 31 00:15:20.216972 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.216855 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129ee960-8884-4992-8468-053b10cbe557-kube-api-access-b4hdr\" (UniqueName: \"kubernetes.io/projected/129ee960-8884-4992-8468-053b10cbe557-kube-api-access-b4hdr\") pod \"129ee960-8884-4992-8468-053b10cbe557\" (UID: \"129ee960-8884-4992-8468-053b10cbe557\") " Oct 31 00:15:20.217117 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.217096 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129ee960-8884-4992-8468-053b10cbe557-maas-api-service-ca" pod "129ee960-8884-4992-8468-053b10cbe557" (UID: "129ee960-8884-4992-8468-053b10cbe557"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:20.219119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.219097 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129ee960-8884-4992-8468-053b10cbe557-kube-api-access-b4hdr" pod "129ee960-8884-4992-8468-053b10cbe557" (UID: "129ee960-8884-4992-8468-053b10cbe557"). InnerVolumeSpecName "kube-api-access-b4hdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:20.318394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.318317 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129ee960-8884-4992-8468-053b10cbe557-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:15:20.318394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.318355 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4hdr\" (UniqueName: \"kubernetes.io/projected/129ee960-8884-4992-8468-053b10cbe557-kube-api-access-b4hdr\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:15:20.903488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.903457 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" Oct 31 00:15:20.903488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.903472 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv" event={"ID":"129ee960-8884-4992-8468-053b10cbe557","Type":"ContainerDied","Data":"5f95aeb4a70fd92c41aa9bcaaf681bb664a5eabe2d5fb1d6539cf85a120ecb8c"} Oct 31 00:15:20.903780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:20.903538 4969 scope.go:122] "RemoveContainer" containerID="635fef01cda42e148a4231224e6a3e1ea3e47f6b398d82d86fe1f0bb7b6dc649" Oct 31 00:15:21.068674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:21.068645 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv"] Oct 31 00:15:21.075513 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:21.075489 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-27pmv"] Oct 31 00:15:21.229298 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:21.229220 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:21.241947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:21.241925 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:21.732925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:21.732883 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129ee960-8884-4992-8468-053b10cbe557" path="/var/lib/kubelet/pods/129ee960-8884-4992-8468-053b10cbe557/volumes" Oct 31 00:15:21.921820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:21.921791 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck" Oct 31 00:15:23.523237 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.523206 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw"] Oct 31 00:15:23.523820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.523800 4969 cpu_manager.go:414] "containerMap: removing container" podUID="129ee960-8884-4992-8468-053b10cbe557" containerName="cleanup" Oct 31 00:15:23.523909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.523822 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129ee960-8884-4992-8468-053b10cbe557" containerName="cleanup" Oct 31 00:15:23.523909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.523839 4969 cpu_manager.go:414] "containerMap: removing container" podUID="129ee960-8884-4992-8468-053b10cbe557" containerName="cleanup" Oct 31 00:15:23.523909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.523846 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129ee960-8884-4992-8468-053b10cbe557" containerName="cleanup" Oct 31 00:15:23.524057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.523963 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="129ee960-8884-4992-8468-053b10cbe557" containerName="cleanup" Oct 31 00:15:23.524057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.523997 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="129ee960-8884-4992-8468-053b10cbe557" containerName="cleanup" Oct 31 00:15:23.528491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.528467 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.586246 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.586212 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw"] Oct 31 00:15:23.632321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.632292 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 00:15:23.646811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.646776 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwls\" (UniqueName: \"kubernetes.io/projected/f8736325-6dcb-4b95-ac2c-a9a25635af0a-kube-api-access-dpwls\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.646951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.646820 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.646951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.646903 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.646951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.646924 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8736325-6dcb-4b95-ac2c-a9a25635af0a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.647136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.647028 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.647136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.647063 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.748370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.748339 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.748370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.748371 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8736325-6dcb-4b95-ac2c-a9a25635af0a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.748660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.748415 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.748660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.748443 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.748660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.748467 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwls\" (UniqueName: \"kubernetes.io/projected/f8736325-6dcb-4b95-ac2c-a9a25635af0a-kube-api-access-dpwls\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.748660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.748498 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.748867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.748840 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.748911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.748884 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.748911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.748902 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.750757 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.750734 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8736325-6dcb-4b95-ac2c-a9a25635af0a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.751153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.751133 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8736325-6dcb-4b95-ac2c-a9a25635af0a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.809396 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.809324 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwls\" (UniqueName: \"kubernetes.io/projected/f8736325-6dcb-4b95-ac2c-a9a25635af0a-kube-api-access-dpwls\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw\" (UID: \"f8736325-6dcb-4b95-ac2c-a9a25635af0a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:23.840150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:23.840119 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:24.223632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:24.223587 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw"] Oct 31 00:15:24.226264 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:15:24.226237 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8736325_6dcb_4b95_ac2c_a9a25635af0a.slice/crio-7b904c988a55b98c337bdcca19c2722034aeb078afdc34c179366033e4af4291 WatchSource:0}: Error finding container 7b904c988a55b98c337bdcca19c2722034aeb078afdc34c179366033e4af4291: Status 404 returned error can't find the container with id 7b904c988a55b98c337bdcca19c2722034aeb078afdc34c179366033e4af4291 Oct 31 00:15:24.931293 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:24.931258 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" event={"ID":"f8736325-6dcb-4b95-ac2c-a9a25635af0a","Type":"ContainerStarted","Data":"a7d322f7ad2899beb095fe6890fb318d823d70b78cc0e31d840ba8dff824af23"} Oct 31 00:15:24.931656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:24.931309 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" event={"ID":"f8736325-6dcb-4b95-ac2c-a9a25635af0a","Type":"ContainerStarted","Data":"7b904c988a55b98c337bdcca19c2722034aeb078afdc34c179366033e4af4291"} Oct 31 00:15:25.051675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:25.051623 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podStartSLOduration=1.470781109 podStartE2EDuration="2.051607989s" totalImagesPullingTime="580.82688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:25.049989792 +0000 UTC m=+757.525348214" watchObservedRunningTime="2026-10-31 00:15:25.051607989 +0000 UTC m=+757.526966406" Oct 31 00:15:25.840468 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:25.840436 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:32.727045 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:15:32.727007 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:15:33.840990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:33.840955 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:33.853316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:33.853290 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:33.997327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:33.997302 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:15:41.498761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.498730 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc"] Oct 31 00:15:41.499212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.499112 4969 cpu_manager.go:414] "containerMap: removing container" podUID="129ee960-8884-4992-8468-053b10cbe557" containerName="cleanup" Oct 31 00:15:41.499212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.499122 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129ee960-8884-4992-8468-053b10cbe557" containerName="cleanup" Oct 31 00:15:41.499212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.499212 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="129ee960-8884-4992-8468-053b10cbe557" containerName="cleanup" Oct 31 00:15:41.506417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.506396 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.612802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.612766 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.612802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.612803 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.613019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.612827 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.613019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.612980 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.613087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.613067 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.613122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.613090 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.613156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.613138 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.613188 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.613160 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.613188 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.613178 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lj86\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-kube-api-access-7lj86\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.714673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.714625 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.714673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.714677 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.714673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.714710 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.714673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.714741 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.714673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.714772 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lj86\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-kube-api-access-7lj86\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.714673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.714828 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.715278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.714855 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.715278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.714883 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.715278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.714938 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.715278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.715022 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc"] Oct 31 00:15:41.715278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.715247 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.716213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.715284 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.716213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.715348 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.716213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.715977 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.716538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.716515 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.718172 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.718148 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.718461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.718432 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.776790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.776720 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.777063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.777041 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lj86\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-kube-api-access-7lj86\") pod \"maas-default-gateway-openshift-default-58c6585447-kf9dc\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:41.820292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:41.820270 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:42.248830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:42.248801 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc"] Oct 31 00:15:42.250284 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:15:42.250254 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4951526d_be25_4f2a_8c1a_ff8a4130ca07.slice/crio-a28555e08b48f911b6b852efb505d7957acb3628d693d264e67d875ccf7f354f WatchSource:0}: Error finding container a28555e08b48f911b6b852efb505d7957acb3628d693d264e67d875ccf7f354f: Status 404 returned error can't find the container with id a28555e08b48f911b6b852efb505d7957acb3628d693d264e67d875ccf7f354f Oct 31 00:15:42.252480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:42.252451 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:15:42.252528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:42.252515 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:15:42.252563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:42.252545 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:15:43.037838 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:43.037799 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" event={"ID":"4951526d-be25-4f2a-8c1a-ff8a4130ca07","Type":"ContainerStarted","Data":"e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b"} Oct 31 00:15:43.038254 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:43.037884 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" event={"ID":"4951526d-be25-4f2a-8c1a-ff8a4130ca07","Type":"ContainerStarted","Data":"a28555e08b48f911b6b852efb505d7957acb3628d693d264e67d875ccf7f354f"} Oct 31 00:15:43.821060 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:43.821027 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:43.822613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:43.822557 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:15:43.822744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:43.822627 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:15:44.821335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:44.821246 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:15:44.821335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:44.821301 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:15:45.821041 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:45.821006 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:15:45.821216 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:45.821058 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:15:46.820994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:46.820959 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:15:46.821393 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:46.821016 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:15:47.820679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:47.820649 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:15:47.820803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:47.820712 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:15:48.821266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:48.821225 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-kf9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:15:48.821681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:48.821309 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:15:50.507574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:50.507541 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:50.508061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:50.507887 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:50.508925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:50.508905 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:15:50.727792 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:15:50.727735 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" podStartSLOduration=9.727722086 podStartE2EDuration="9.727722086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:43.259086746 +0000 UTC m=+775.734445169" watchObservedRunningTime="2026-10-31 00:15:50.727722086 +0000 UTC m=+783.203080506" Oct 31 00:16:02.727500 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:16:02.727462 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:16:25.738181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.738150 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2"] Oct 31 00:16:25.741766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.741750 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.794107 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.794068 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2"] Oct 31 00:16:25.810855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.810829 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istiod-ca-cert\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.811038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.810877 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz5vt\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-kube-api-access-wz5vt\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.811038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.810906 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-certs\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.811038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.810984 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-credential-socket\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.811228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.811190 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-envoy\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.811291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.811226 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-data\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.811291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.811268 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-token\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.811404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.811296 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-socket\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.811404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.811397 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-podinfo\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.840390 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.840364 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0e1bbe4b-openshift-default-dockercfg-xgzpz\"" Oct 31 00:16:25.912405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912369 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-envoy\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912406 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-data\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912439 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-token\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912465 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-socket\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912516 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-podinfo\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912567 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istiod-ca-cert\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912625 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz5vt\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-kube-api-access-wz5vt\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912650 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-certs\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912681 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-credential-socket\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912847 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-socket\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.912973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.912867 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-data\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.913125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.913073 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-certs\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.913125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.913089 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-credential-socket\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.913488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.913468 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istiod-ca-cert\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.915078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.915056 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-envoy\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.915665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.915647 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-podinfo\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.974507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.974475 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-token\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:25.975110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:25.975085 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz5vt\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-kube-api-access-wz5vt\") pod \"e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:26.038542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.038469 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9"] Oct 31 00:16:26.043360 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.043340 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.056336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.056315 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:26.096678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.096632 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-15eab294-openshift-default-dockercfg-blf5n\"" Oct 31 00:16:26.116322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.116280 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9"] Oct 31 00:16:26.116901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.116866 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-socket\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.117048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.116911 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-envoy\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.117048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.116983 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-token\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.117178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.117132 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdq6q\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-kube-api-access-fdq6q\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.117235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.117173 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istiod-ca-cert\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.117290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.117266 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-data\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.117348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.117302 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-certs\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.117348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.117327 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-podinfo\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.117460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.117366 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-credential-socket\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218132 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218097 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-data\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218139 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-certs\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218167 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-podinfo\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218200 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-credential-socket\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218254 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-socket\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218284 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-envoy\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218319 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-token\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218387 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdq6q\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-kube-api-access-fdq6q\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218423 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istiod-ca-cert\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218571 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-data\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218783 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-certs\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218803 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-credential-socket\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.218927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.218885 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-socket\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.219230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.219212 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istiod-ca-cert\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.220836 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.220814 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-envoy\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.221019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.221000 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-podinfo\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.279573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.279531 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-token\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.279830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.279805 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdq6q\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-kube-api-access-fdq6q\") pod \"e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.357114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.357035 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:26.499762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.499734 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2"] Oct 31 00:16:26.501256 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:16:26.501229 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a176f17_fe9b_4996_b2e5_cb715d4c5bec.slice/crio-bd51220089c64823929f4f92db4ec97de92f54806100ab9be8e5889de54c774a WatchSource:0}: Error finding container bd51220089c64823929f4f92db4ec97de92f54806100ab9be8e5889de54c774a: Status 404 returned error can't find the container with id bd51220089c64823929f4f92db4ec97de92f54806100ab9be8e5889de54c774a Oct 31 00:16:26.503340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.503309 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:26.503407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.503389 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:26.503444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.503425 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:26.654314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.654279 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr"] Oct 31 00:16:26.658482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.658463 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.710884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.710855 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4b78c516-gw-openshift-default-dockercfg-hwzb9\"" Oct 31 00:16:26.717710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.717676 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr"] Oct 31 00:16:26.723831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.723798 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-socket\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.724004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.723928 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-envoy\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.724068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.723997 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-podinfo\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.724139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.724116 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5v9s\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-kube-api-access-w5v9s\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.724203 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.724164 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-credential-socket\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.724853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.724812 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-data\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.724959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.724885 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-token\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.725008 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.724952 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istiod-ca-cert\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.725076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.725052 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-certs\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.742833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.742135 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9"] Oct 31 00:16:26.743134 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:16:26.743073 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f907a2_82d5_44ce_8d56_5c3de82f6b41.slice/crio-17cd34ccfa14ce7a6b60674b78ab4f255ee4d25fa0144a4ee8c09afb45ef193a WatchSource:0}: Error finding container 17cd34ccfa14ce7a6b60674b78ab4f255ee4d25fa0144a4ee8c09afb45ef193a: Status 404 returned error can't find the container with id 17cd34ccfa14ce7a6b60674b78ab4f255ee4d25fa0144a4ee8c09afb45ef193a Oct 31 00:16:26.746031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.745998 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:26.746096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.746079 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:26.746140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.746126 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:26.826405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826377 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-data\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826430 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-token\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826456 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istiod-ca-cert\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826492 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-certs\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826547 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-socket\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826643 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-envoy\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826705 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-podinfo\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826756 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5v9s\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-kube-api-access-w5v9s\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826764 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-data\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826818 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-credential-socket\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.826920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.826865 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-certs\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.827346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.827322 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istiod-ca-cert\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.827551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.827529 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-socket\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.827551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.827540 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-credential-socket\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.829547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.829521 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-podinfo\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.829660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.829633 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-envoy\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.902606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.902554 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5v9s\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-kube-api-access-w5v9s\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.903759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.903730 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-token\") pod \"e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:26.971034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:26.970943 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:27.297456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.297364 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" event={"ID":"d7f907a2-82d5-44ce-8d56-5c3de82f6b41","Type":"ContainerStarted","Data":"52c05a964c090d5bc554704dc5ecfa77f82b2c24fc4d157355fdec38dd524874"} Oct 31 00:16:27.297456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.297422 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" event={"ID":"d7f907a2-82d5-44ce-8d56-5c3de82f6b41","Type":"ContainerStarted","Data":"17cd34ccfa14ce7a6b60674b78ab4f255ee4d25fa0144a4ee8c09afb45ef193a"} Oct 31 00:16:27.299063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.299036 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" event={"ID":"2a176f17-fe9b-4996-b2e5-cb715d4c5bec","Type":"ContainerStarted","Data":"1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160"} Oct 31 00:16:27.299176 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.299080 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" event={"ID":"2a176f17-fe9b-4996-b2e5-cb715d4c5bec","Type":"ContainerStarted","Data":"bd51220089c64823929f4f92db4ec97de92f54806100ab9be8e5889de54c774a"} Oct 31 00:16:27.355027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.355004 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr"] Oct 31 00:16:27.356731 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:16:27.356706 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3ee639_6d22_4ce1_907a_0cc21f393eb5.slice/crio-13c6dcbbeecd17a540eae0ca6a0d4d25021cee3d91b30b2d6ac06ba69b4086c5 WatchSource:0}: Error finding container 13c6dcbbeecd17a540eae0ca6a0d4d25021cee3d91b30b2d6ac06ba69b4086c5: Status 404 returned error can't find the container with id 13c6dcbbeecd17a540eae0ca6a0d4d25021cee3d91b30b2d6ac06ba69b4086c5 Oct 31 00:16:27.357485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.357460 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:27.359029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.359000 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:27.359110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.359059 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:27.359110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.359088 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:27.363112 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.363092 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:27.423971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.423929 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" podStartSLOduration=2.423915009 podStartE2EDuration="2.423915009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:27.421223903 +0000 UTC m=+819.896582327" watchObservedRunningTime="2026-10-31 00:16:27.423915009 +0000 UTC m=+819.899273431" Oct 31 00:16:27.674898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:27.674849 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" podStartSLOduration=2.674835103 podStartE2EDuration="2.674835103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:27.673038297 +0000 UTC m=+820.148396726" watchObservedRunningTime="2026-10-31 00:16:27.674835103 +0000 UTC m=+820.150193528" Oct 31 00:16:28.057036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.056955 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:28.062180 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.062156 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:28.307311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.307228 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" event={"ID":"3d3ee639-6d22-4ce1-907a-0cc21f393eb5","Type":"ContainerStarted","Data":"9f66d3bc62f49f227f6a675daa02e7e3d13d31e89ff5bcc68a0262a8c491590a"} Oct 31 00:16:28.307311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.307278 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" event={"ID":"3d3ee639-6d22-4ce1-907a-0cc21f393eb5","Type":"ContainerStarted","Data":"13c6dcbbeecd17a540eae0ca6a0d4d25021cee3d91b30b2d6ac06ba69b4086c5"} Oct 31 00:16:28.307735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.307705 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:28.307848 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.307755 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:28.308418 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.308398 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:16:28.308518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.308447 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:28.431583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.431532 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" podStartSLOduration=2.431517425 podStartE2EDuration="2.431517425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:28.430348127 +0000 UTC m=+820.905706550" watchObservedRunningTime="2026-10-31 00:16:28.431517425 +0000 UTC m=+820.906875848" Oct 31 00:16:28.972069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.972033 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:28.977064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:28.977036 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:29.313161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:29.313073 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:29.313897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:29.313878 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:16:32.726421 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:16:32.726384 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:16:33.506976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.506942 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx"] Oct 31 00:16:33.510892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.510873 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.596964 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.596912 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-envoy\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.596964 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.596973 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468c0ca1-e5de-4634-867b-017cb193eb36-istiod-ca-cert\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.597212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.597084 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-socket\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.597212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.597180 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-data\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.597303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.597227 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468c0ca1-e5de-4634-867b-017cb193eb36-istio-podinfo\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.597303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.597263 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-certs\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.597377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.597305 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5rp\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-kube-api-access-qm5rp\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.597410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.597384 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-istio-token\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.597410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.597403 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-credential-socket\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.615960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.615926 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b83a178e-openshift-default-dockercfg-ck7rj\"" Oct 31 00:16:33.619231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.619201 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx"] Oct 31 00:16:33.697907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.697879 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-data\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.697923 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468c0ca1-e5de-4634-867b-017cb193eb36-istio-podinfo\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.697954 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-certs\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.697979 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5rp\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-kube-api-access-qm5rp\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698017 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-istio-token\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698042 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-credential-socket\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698347 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698093 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-envoy\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698347 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698145 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468c0ca1-e5de-4634-867b-017cb193eb36-istiod-ca-cert\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698347 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698192 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-socket\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698340 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-data\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698433 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-credential-socket\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698523 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-socket\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698658 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-certs\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.698974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.698848 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468c0ca1-e5de-4634-867b-017cb193eb36-istiod-ca-cert\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.700680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.700654 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-envoy\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.700766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.700752 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468c0ca1-e5de-4634-867b-017cb193eb36-istio-podinfo\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.759305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.759230 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-istio-token\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.759672 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.759386 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5rp\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-kube-api-access-qm5rp\") pod \"e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:33.824909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:33.824883 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:34.258469 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:34.258440 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx"] Oct 31 00:16:34.262921 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:16:34.262887 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468c0ca1_e5de_4634_867b_017cb193eb36.slice/crio-5d2274a7599fad23621db8c65c82239d2d9bbd1f7f77dc8eedd64c27b0d425ab WatchSource:0}: Error finding container 5d2274a7599fad23621db8c65c82239d2d9bbd1f7f77dc8eedd64c27b0d425ab: Status 404 returned error can't find the container with id 5d2274a7599fad23621db8c65c82239d2d9bbd1f7f77dc8eedd64c27b0d425ab Oct 31 00:16:34.266908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:34.266876 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:34.267000 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:34.266944 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:34.267000 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:34.266986 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:34.344807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:34.344777 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" event={"ID":"468c0ca1-e5de-4634-867b-017cb193eb36","Type":"ContainerStarted","Data":"c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5"} Oct 31 00:16:34.344934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:34.344829 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" event={"ID":"468c0ca1-e5de-4634-867b-017cb193eb36","Type":"ContainerStarted","Data":"5d2274a7599fad23621db8c65c82239d2d9bbd1f7f77dc8eedd64c27b0d425ab"} Oct 31 00:16:34.470825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:34.470776 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" podStartSLOduration=1.470760565 podStartE2EDuration="1.470760565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:34.466849198 +0000 UTC m=+826.942207620" watchObservedRunningTime="2026-10-31 00:16:34.470760565 +0000 UTC m=+826.946118987" Oct 31 00:16:34.825821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:34.825794 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:35.830841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:35.830813 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:36.356658 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:36.356627 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:36.357646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:36.357626 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:16:38.979742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:38.979709 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d"] Oct 31 00:16:38.983664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:38.983641 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:39.044276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.044249 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d"] Oct 31 00:16:39.052602 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.052560 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e52a4d-c6f6-4b36-804b-b2911893dc2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d\" (UID: \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:39.052726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.052672 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgkrt\" (UniqueName: \"kubernetes.io/projected/28e52a4d-c6f6-4b36-804b-b2911893dc2c-kube-api-access-qgkrt\") pod \"payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d\" (UID: \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:39.089527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.089499 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-15eab294\"" Oct 31 00:16:39.119654 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.119634 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-15eab294-dockercfg-f6jlv\"" Oct 31 00:16:39.151785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.151751 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49"] Oct 31 00:16:39.154492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.154463 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e52a4d-c6f6-4b36-804b-b2911893dc2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d\" (UID: \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:39.154671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.154533 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgkrt\" (UniqueName: \"kubernetes.io/projected/28e52a4d-c6f6-4b36-804b-b2911893dc2c-kube-api-access-qgkrt\") pod \"payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d\" (UID: \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:39.155617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.155574 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:39.156092 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.156068 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e52a4d-c6f6-4b36-804b-b2911893dc2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d\" (UID: \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:39.207085 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.207057 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49"] Oct 31 00:16:39.215945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.215919 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgkrt\" (UniqueName: \"kubernetes.io/projected/28e52a4d-c6f6-4b36-804b-b2911893dc2c-kube-api-access-qgkrt\") pod \"payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d\" (UID: \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:39.255879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.255818 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddzfj\" (UniqueName: \"kubernetes.io/projected/9d08a915-f585-4694-b8f7-447e8739abb7-kube-api-access-ddzfj\") pod \"payload-processing-e2e-ait-15eab294-6db845f85-gzp49\" (UID: \"9d08a915-f585-4694-b8f7-447e8739abb7\") " pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:39.255879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.255865 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d08a915-f585-4694-b8f7-447e8739abb7-plugins-config-volume\") pod \"payload-processing-e2e-ait-15eab294-6db845f85-gzp49\" (UID: \"9d08a915-f585-4694-b8f7-447e8739abb7\") " pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:39.297513 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.297483 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:39.360804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.357620 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddzfj\" (UniqueName: \"kubernetes.io/projected/9d08a915-f585-4694-b8f7-447e8739abb7-kube-api-access-ddzfj\") pod \"payload-processing-e2e-ait-15eab294-6db845f85-gzp49\" (UID: \"9d08a915-f585-4694-b8f7-447e8739abb7\") " pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:39.360804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.357690 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d08a915-f585-4694-b8f7-447e8739abb7-plugins-config-volume\") pod \"payload-processing-e2e-ait-15eab294-6db845f85-gzp49\" (UID: \"9d08a915-f585-4694-b8f7-447e8739abb7\") " pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:39.360804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.359146 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d08a915-f585-4694-b8f7-447e8739abb7-plugins-config-volume\") pod \"payload-processing-e2e-ait-15eab294-6db845f85-gzp49\" (UID: \"9d08a915-f585-4694-b8f7-447e8739abb7\") " pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:39.418537 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.418497 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddzfj\" (UniqueName: \"kubernetes.io/projected/9d08a915-f585-4694-b8f7-447e8739abb7-kube-api-access-ddzfj\") pod \"payload-processing-e2e-ait-15eab294-6db845f85-gzp49\" (UID: \"9d08a915-f585-4694-b8f7-447e8739abb7\") " pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:39.474731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.474699 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:39.674826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.674800 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d"] Oct 31 00:16:39.676153 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:16:39.676129 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e52a4d_c6f6_4b36_804b_b2911893dc2c.slice/crio-dc4159609fee3dc15117e987faa660ba7943bbde63b2577a7339aef3db193441 WatchSource:0}: Error finding container dc4159609fee3dc15117e987faa660ba7943bbde63b2577a7339aef3db193441: Status 404 returned error can't find the container with id dc4159609fee3dc15117e987faa660ba7943bbde63b2577a7339aef3db193441 Oct 31 00:16:39.854689 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:39.854661 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49"] Oct 31 00:16:39.855859 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:16:39.855836 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d08a915_f585_4694_b8f7_447e8739abb7.slice/crio-fc50488de2b6358cb093da1d35daa6f5696ae417c23653fff96337bfface114a WatchSource:0}: Error finding container fc50488de2b6358cb093da1d35daa6f5696ae417c23653fff96337bfface114a: Status 404 returned error can't find the container with id fc50488de2b6358cb093da1d35daa6f5696ae417c23653fff96337bfface114a Oct 31 00:16:40.380859 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:40.380826 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" event={"ID":"9d08a915-f585-4694-b8f7-447e8739abb7","Type":"ContainerStarted","Data":"fc50488de2b6358cb093da1d35daa6f5696ae417c23653fff96337bfface114a"} Oct 31 00:16:40.382112 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:40.382089 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" event={"ID":"28e52a4d-c6f6-4b36-804b-b2911893dc2c","Type":"ContainerStarted","Data":"dc4159609fee3dc15117e987faa660ba7943bbde63b2577a7339aef3db193441"} Oct 31 00:16:41.581684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.581644 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7"] Oct 31 00:16:41.585865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.585838 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:41.644882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.644853 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7"] Oct 31 00:16:41.686451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.686131 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7\" (UID: \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:41.686451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.686224 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9fx\" (UniqueName: \"kubernetes.io/projected/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-kube-api-access-4g9fx\") pod \"payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7\" (UID: \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:41.690502 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.690329 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e1bbe4b\"" Oct 31 00:16:41.690502 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.690469 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e1bbe4b-dockercfg-ntjbq\"" Oct 31 00:16:41.787814 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.787523 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7\" (UID: \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:41.787814 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.787578 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9fx\" (UniqueName: \"kubernetes.io/projected/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-kube-api-access-4g9fx\") pod \"payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7\" (UID: \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:41.788379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.788352 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7\" (UID: \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:41.864837 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.864725 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9fx\" (UniqueName: \"kubernetes.io/projected/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-kube-api-access-4g9fx\") pod \"payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7\" (UID: \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:41.903211 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:41.903178 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:42.304045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:42.304015 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7"] Oct 31 00:16:42.306383 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:16:42.306352 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad60ac5_4a7f_4cc9_9633_79bcbf65ecfb.slice/crio-b20f855d68e7cd45b8bee5fb5f3e6ad85032f7cc48712cb8ba5adc1d675bdee5 WatchSource:0}: Error finding container b20f855d68e7cd45b8bee5fb5f3e6ad85032f7cc48712cb8ba5adc1d675bdee5: Status 404 returned error can't find the container with id b20f855d68e7cd45b8bee5fb5f3e6ad85032f7cc48712cb8ba5adc1d675bdee5 Oct 31 00:16:42.400451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:42.400372 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" event={"ID":"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb","Type":"ContainerStarted","Data":"b20f855d68e7cd45b8bee5fb5f3e6ad85032f7cc48712cb8ba5adc1d675bdee5"} Oct 31 00:16:43.407753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:43.407623 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" event={"ID":"9d08a915-f585-4694-b8f7-447e8739abb7","Type":"ContainerStarted","Data":"0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb"} Oct 31 00:16:43.407753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:43.407687 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:43.409240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:43.409217 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" event={"ID":"28e52a4d-c6f6-4b36-804b-b2911893dc2c","Type":"ContainerStarted","Data":"609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da"} Oct 31 00:16:43.409373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:43.409286 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:43.410765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:43.410744 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" event={"ID":"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb","Type":"ContainerStarted","Data":"f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9"} Oct 31 00:16:43.410845 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:43.410797 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:43.530774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:43.527119 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" podStartSLOduration=1.52478701 podStartE2EDuration="4.527101687s" totalImagesPullingTime="3.002314677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:43.523136072 +0000 UTC m=+835.998494498" watchObservedRunningTime="2026-10-31 00:16:43.527101687 +0000 UTC m=+836.002460112" Oct 31 00:16:43.644564 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:43.644498 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" podStartSLOduration=1.830220419 podStartE2EDuration="2.644480507s" totalImagesPullingTime="814.260088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:43.641360435 +0000 UTC m=+836.116718859" watchObservedRunningTime="2026-10-31 00:16:43.644480507 +0000 UTC m=+836.119838931" Oct 31 00:16:43.759440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:43.759341 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" podStartSLOduration=2.610078878 podStartE2EDuration="5.759327161s" totalImagesPullingTime="3.149248283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:43.758118354 +0000 UTC m=+836.233476777" watchObservedRunningTime="2026-10-31 00:16:43.759327161 +0000 UTC m=+836.234685584" Oct 31 00:16:44.226474 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.226414 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz"] Oct 31 00:16:44.231410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.231386 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:44.283552 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.283523 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6ab60e97\"" Oct 31 00:16:44.283851 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.283822 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6ab60e97-dockercfg-854wx\"" Oct 31 00:16:44.287739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.287713 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz"] Oct 31 00:16:44.316181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.316155 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbp8\" (UniqueName: \"kubernetes.io/projected/486a00d1-f042-4238-9b34-52552778e7ac-kube-api-access-mpbp8\") pod \"payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz\" (UID: \"486a00d1-f042-4238-9b34-52552778e7ac\") " pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:44.316293 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.316262 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486a00d1-f042-4238-9b34-52552778e7ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz\" (UID: \"486a00d1-f042-4238-9b34-52552778e7ac\") " pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:44.416908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.416884 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbp8\" (UniqueName: \"kubernetes.io/projected/486a00d1-f042-4238-9b34-52552778e7ac-kube-api-access-mpbp8\") pod \"payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz\" (UID: \"486a00d1-f042-4238-9b34-52552778e7ac\") " pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:44.417279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.416955 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486a00d1-f042-4238-9b34-52552778e7ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz\" (UID: \"486a00d1-f042-4238-9b34-52552778e7ac\") " pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:44.417629 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.417578 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486a00d1-f042-4238-9b34-52552778e7ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz\" (UID: \"486a00d1-f042-4238-9b34-52552778e7ac\") " pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:44.478166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.478096 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbp8\" (UniqueName: \"kubernetes.io/projected/486a00d1-f042-4238-9b34-52552778e7ac-kube-api-access-mpbp8\") pod \"payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz\" (UID: \"486a00d1-f042-4238-9b34-52552778e7ac\") " pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:44.545327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.545295 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:44.920816 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:44.920789 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz"] Oct 31 00:16:44.922320 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:16:44.922296 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486a00d1_f042_4238_9b34_52552778e7ac.slice/crio-70f77b6fb2930ed123d96b709c32c23e4136b288d7d94682cedd1c4a9e07fa2b WatchSource:0}: Error finding container 70f77b6fb2930ed123d96b709c32c23e4136b288d7d94682cedd1c4a9e07fa2b: Status 404 returned error can't find the container with id 70f77b6fb2930ed123d96b709c32c23e4136b288d7d94682cedd1c4a9e07fa2b Oct 31 00:16:45.425373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:45.425332 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" event={"ID":"486a00d1-f042-4238-9b34-52552778e7ac","Type":"ContainerStarted","Data":"ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217"} Oct 31 00:16:45.425897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:45.425394 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" event={"ID":"486a00d1-f042-4238-9b34-52552778e7ac","Type":"ContainerStarted","Data":"70f77b6fb2930ed123d96b709c32c23e4136b288d7d94682cedd1c4a9e07fa2b"} Oct 31 00:16:45.425897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:45.425463 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:45.543284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:45.543235 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" podStartSLOduration=1.54322164 podStartE2EDuration="1.54322164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:45.54147047 +0000 UTC m=+838.016828894" watchObservedRunningTime="2026-10-31 00:16:45.54322164 +0000 UTC m=+838.018580060" Oct 31 00:16:54.419617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:54.419569 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:16:54.420108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:54.419755 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:16:54.420108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:54.419817 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:16:55.508407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:55.508376 4969 generic.go:356] "Generic (PLEG): container finished" podID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerID="1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160" exitCode=0 Oct 31 00:16:55.508870 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:55.508459 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" event={"ID":"2a176f17-fe9b-4996-b2e5-cb715d4c5bec","Type":"ContainerDied","Data":"1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160"} Oct 31 00:16:55.508911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:55.508899 4969 scope.go:122] "RemoveContainer" containerID="1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160" Oct 31 00:16:55.510214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:55.510183 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:55.510304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:55.510253 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:55.510304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:55.510286 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:56.434401 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:56.434371 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:16:56.535112 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:56.535079 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" event={"ID":"2a176f17-fe9b-4996-b2e5-cb715d4c5bec","Type":"ContainerStarted","Data":"6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2"} Oct 31 00:16:57.057031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:57.056996 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:57.058074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:57.058042 4969 patch_prober.go:29] interesting pod/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:16:57.058134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:57.058109 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:16:58.057085 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:58.057049 4969 patch_prober.go:29] interesting pod/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:16:58.057481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:58.057113 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:16:58.315708 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:58.314334 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:16:59.056902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:59.056870 4969 patch_prober.go:29] interesting pod/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:16:59.057091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:16:59.056918 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:17:00.057097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:00.057062 4969 patch_prober.go:29] interesting pod/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:17:00.057486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:00.057123 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:17:01.057515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:01.057432 4969 patch_prober.go:29] interesting pod/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:17:01.057515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:01.057505 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:17:02.079566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:02.078870 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:17:02.081764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:02.081203 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:17:02.727439 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:17:02.727403 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:17:04.588364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:04.588274 4969 generic.go:356] "Generic (PLEG): container finished" podID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerID="9f66d3bc62f49f227f6a675daa02e7e3d13d31e89ff5bcc68a0262a8c491590a" exitCode=0 Oct 31 00:17:04.588364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:04.588348 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" event={"ID":"3d3ee639-6d22-4ce1-907a-0cc21f393eb5","Type":"ContainerDied","Data":"9f66d3bc62f49f227f6a675daa02e7e3d13d31e89ff5bcc68a0262a8c491590a"} Oct 31 00:17:04.588840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:04.588808 4969 scope.go:122] "RemoveContainer" containerID="9f66d3bc62f49f227f6a675daa02e7e3d13d31e89ff5bcc68a0262a8c491590a" Oct 31 00:17:04.590081 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:04.590054 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:04.590133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:04.590124 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:04.590167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:04.590155 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:05.632551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:05.632508 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" event={"ID":"3d3ee639-6d22-4ce1-907a-0cc21f393eb5","Type":"ContainerStarted","Data":"73d49eb4839d17bea94e49f947fe5e12765336693b65336382e3d8aff0068cdf"} Oct 31 00:17:05.971518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:05.971434 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:17:05.971707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:05.971578 4969 patch_prober.go:29] interesting pod/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:17:05.971707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:05.971643 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:17:06.972007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:06.971974 4969 patch_prober.go:29] interesting pod/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:17:06.972389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:06.972030 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:17:07.972199 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:07.972160 4969 patch_prober.go:29] interesting pod/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:17:07.972609 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:07.972222 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:17:08.971842 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:08.971806 4969 patch_prober.go:29] interesting pod/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:17:08.972047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:08.971864 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:17:09.972252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:09.972222 4969 patch_prober.go:29] interesting pod/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:17:09.972671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:09.972288 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:17:10.975510 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:10.975481 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:17:10.975979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:10.975961 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:17:10.976672 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:10.976651 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:17:13.834796 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:13.834583 4969 patch_prober.go:29] interesting pod/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:60550->10.133.0.59:15021: read: connection reset by peer" start-of-body= Oct 31 00:17:13.834796 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:13.834680 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:60550->10.133.0.59:15021: read: connection reset by peer" Oct 31 00:17:14.710064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:14.710030 4969 generic.go:356] "Generic (PLEG): container finished" podID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerID="52c05a964c090d5bc554704dc5ecfa77f82b2c24fc4d157355fdec38dd524874" exitCode=0 Oct 31 00:17:14.710242 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:14.710058 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" event={"ID":"d7f907a2-82d5-44ce-8d56-5c3de82f6b41","Type":"ContainerDied","Data":"52c05a964c090d5bc554704dc5ecfa77f82b2c24fc4d157355fdec38dd524874"} Oct 31 00:17:14.710411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:14.710398 4969 scope.go:122] "RemoveContainer" containerID="52c05a964c090d5bc554704dc5ecfa77f82b2c24fc4d157355fdec38dd524874" Oct 31 00:17:14.711553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:14.711529 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:14.711641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:14.711624 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:14.711691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:14.711673 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:15.742899 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:15.742868 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" event={"ID":"d7f907a2-82d5-44ce-8d56-5c3de82f6b41","Type":"ContainerStarted","Data":"978465d2019fa539b8ae78c61a671d84a4f6c9e0271965643d4a690c333ae113"} Oct 31 00:17:16.357767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.357730 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:17:16.357944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.357900 4969 patch_prober.go:29] interesting pod/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:17:16.357990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.357942 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:17:16.517617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.516705 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd"] Oct 31 00:17:16.528441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.527934 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.542017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.541992 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.542140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.542036 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.542140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.542120 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67hv\" (UniqueName: \"kubernetes.io/projected/bab19133-b61c-41ee-84c4-3830e2f1bda1-kube-api-access-f67hv\") pod \"maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.563363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.563337 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd"] Oct 31 00:17:16.612671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.612139 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:17:16.612671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.612262 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:17:16.642834 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.642801 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.642994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.642847 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.642994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.642971 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67hv\" (UniqueName: \"kubernetes.io/projected/bab19133-b61c-41ee-84c4-3830e2f1bda1-kube-api-access-f67hv\") pod \"maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.643666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.643641 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.648076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.648054 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.729574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.729528 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67hv\" (UniqueName: \"kubernetes.io/projected/bab19133-b61c-41ee-84c4-3830e2f1bda1-kube-api-access-f67hv\") pod \"maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:16.841097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:16.841062 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:17.320630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:17.299401 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd"] Oct 31 00:17:17.357700 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:17.357677 4969 patch_prober.go:29] interesting pod/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:17:17.357786 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:17.357718 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:17:17.756417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:17.756384 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" event={"ID":"bab19133-b61c-41ee-84c4-3830e2f1bda1","Type":"ContainerStarted","Data":"c8eb34d97938f388ef9518ff9604d37d28158427c45c58a696a9a38dfcf7b565"} Oct 31 00:17:18.357523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:18.357487 4969 patch_prober.go:29] interesting pod/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:17:18.357933 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:18.357535 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:17:18.764002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:18.763970 4969 generic.go:356] "Generic (PLEG): container finished" podID="bab19133-b61c-41ee-84c4-3830e2f1bda1" containerID="375c5e1e113007ba7fe4ecdbafdbe433513b33792d55c0d19a1592edaacdfa7b" exitCode=0 Oct 31 00:17:18.764191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:18.764062 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" event={"ID":"bab19133-b61c-41ee-84c4-3830e2f1bda1","Type":"ContainerDied","Data":"375c5e1e113007ba7fe4ecdbafdbe433513b33792d55c0d19a1592edaacdfa7b"} Oct 31 00:17:19.358011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:19.357975 4969 patch_prober.go:29] interesting pod/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:17:19.358403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:19.358035 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:17:19.967175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:19.967149 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:20.076187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.076111 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-openshift-ca\") pod \"bab19133-b61c-41ee-84c4-3830e2f1bda1\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " Oct 31 00:17:20.076187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.076166 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab19133-b61c-41ee-84c4-3830e2f1bda1-kube-api-access-f67hv\" (UniqueName: \"kubernetes.io/projected/bab19133-b61c-41ee-84c4-3830e2f1bda1-kube-api-access-f67hv\") pod \"bab19133-b61c-41ee-84c4-3830e2f1bda1\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " Oct 31 00:17:20.076357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.076203 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-xks-ca\") pod \"bab19133-b61c-41ee-84c4-3830e2f1bda1\" (UID: \"bab19133-b61c-41ee-84c4-3830e2f1bda1\") " Oct 31 00:17:20.076541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.076511 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-openshift-ca" pod "bab19133-b61c-41ee-84c4-3830e2f1bda1" (UID: "bab19133-b61c-41ee-84c4-3830e2f1bda1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:20.100086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.100033 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-xks-ca" pod "bab19133-b61c-41ee-84c4-3830e2f1bda1" (UID: "bab19133-b61c-41ee-84c4-3830e2f1bda1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:20.106790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.106752 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab19133-b61c-41ee-84c4-3830e2f1bda1-kube-api-access-f67hv" pod "bab19133-b61c-41ee-84c4-3830e2f1bda1" (UID: "bab19133-b61c-41ee-84c4-3830e2f1bda1"). InnerVolumeSpecName "kube-api-access-f67hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:20.201620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.177226 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67hv\" (UniqueName: \"kubernetes.io/projected/bab19133-b61c-41ee-84c4-3830e2f1bda1-kube-api-access-f67hv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:20.201620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.177260 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:20.201620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.177276 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab19133-b61c-41ee-84c4-3830e2f1bda1-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:20.357618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.357511 4969 patch_prober.go:29] interesting pod/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:17:20.357618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.357586 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:17:20.780436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.780398 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" Oct 31 00:17:20.780436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.780432 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd" event={"ID":"bab19133-b61c-41ee-84c4-3830e2f1bda1","Type":"ContainerDied","Data":"c8eb34d97938f388ef9518ff9604d37d28158427c45c58a696a9a38dfcf7b565"} Oct 31 00:17:20.780938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:20.780482 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8eb34d97938f388ef9518ff9604d37d28158427c45c58a696a9a38dfcf7b565" Oct 31 00:17:21.361494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:21.361460 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:17:21.361890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:21.361862 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:17:21.362701 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:21.362677 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:17:22.398732 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:22.398700 4969 patch_prober.go:29] interesting pod/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:22.399122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:22.398763 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:23.801497 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:23.801466 4969 generic.go:356] "Generic (PLEG): container finished" podID="468c0ca1-e5de-4634-867b-017cb193eb36" containerID="c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5" exitCode=0 Oct 31 00:17:23.801949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:23.801513 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" event={"ID":"468c0ca1-e5de-4634-867b-017cb193eb36","Type":"ContainerDied","Data":"c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5"} Oct 31 00:17:23.801949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:23.801927 4969 scope.go:122] "RemoveContainer" containerID="c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5" Oct 31 00:17:23.803244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:23.803214 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:23.803353 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:23.803288 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:23.803353 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:23.803325 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:24.422561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.422526 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp"] Oct 31 00:17:24.423282 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.423259 4969 cpu_manager.go:414] "containerMap: removing container" podUID="bab19133-b61c-41ee-84c4-3830e2f1bda1" containerName="revoke-keys" Oct 31 00:17:24.423330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.423289 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab19133-b61c-41ee-84c4-3830e2f1bda1" containerName="revoke-keys" Oct 31 00:17:24.423489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.423478 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab19133-b61c-41ee-84c4-3830e2f1bda1" containerName="revoke-keys" Oct 31 00:17:24.426019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.425990 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.483257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.482303 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:17:24.483257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.482427 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:17:24.506914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.499876 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp"] Oct 31 00:17:24.537200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.537167 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f78z\" (UniqueName: \"kubernetes.io/projected/eff40f65-1fe2-42b7-8679-c6159b61c320-kube-api-access-2f78z\") pod \"maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.537200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.537212 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.537407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.537328 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.638444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.638404 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f78z\" (UniqueName: \"kubernetes.io/projected/eff40f65-1fe2-42b7-8679-c6159b61c320-kube-api-access-2f78z\") pod \"maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.638444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.638449 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.638757 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.638482 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.639098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.639074 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.641141 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.641118 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.702474 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.702384 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f78z\" (UniqueName: \"kubernetes.io/projected/eff40f65-1fe2-42b7-8679-c6159b61c320-kube-api-access-2f78z\") pod \"maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.739422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.739397 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:24.814143 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.813298 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" event={"ID":"468c0ca1-e5de-4634-867b-017cb193eb36","Type":"ContainerStarted","Data":"c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1"} Oct 31 00:17:24.826727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.826280 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:17:24.826727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.826401 4969 patch_prober.go:29] interesting pod/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.826727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:24.826446 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:17:25.140582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:25.140551 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp"] Oct 31 00:17:25.141805 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:17:25.141784 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff40f65_1fe2_42b7_8679_c6159b61c320.slice/crio-ba1f47190d09be38da84c564f5b585ce3ee6b9aa8e577894bfed0bf645c18aca WatchSource:0}: Error finding container ba1f47190d09be38da84c564f5b585ce3ee6b9aa8e577894bfed0bf645c18aca: Status 404 returned error can't find the container with id ba1f47190d09be38da84c564f5b585ce3ee6b9aa8e577894bfed0bf645c18aca Oct 31 00:17:25.821499 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:25.821406 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" event={"ID":"eff40f65-1fe2-42b7-8679-c6159b61c320","Type":"ContainerStarted","Data":"7215369b1936776748dcddd47e255ef78b061f81c91a6475797dddb1e6ab7f23"} Oct 31 00:17:25.821499 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:25.821464 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" event={"ID":"eff40f65-1fe2-42b7-8679-c6159b61c320","Type":"ContainerStarted","Data":"ba1f47190d09be38da84c564f5b585ce3ee6b9aa8e577894bfed0bf645c18aca"} Oct 31 00:17:25.826221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:25.826197 4969 patch_prober.go:29] interesting pod/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.826314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:25.826250 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:17:25.938642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:25.938573 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" podStartSLOduration=1.938558757 podStartE2EDuration="1.938558757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:25.936252577 +0000 UTC m=+878.411611001" watchObservedRunningTime="2026-10-31 00:17:25.938558757 +0000 UTC m=+878.413917179" Oct 31 00:17:26.825350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:26.825312 4969 patch_prober.go:29] interesting pod/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:17:26.825782 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:26.825364 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:17:26.829839 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:26.829811 4969 generic.go:356] "Generic (PLEG): container finished" podID="eff40f65-1fe2-42b7-8679-c6159b61c320" containerID="7215369b1936776748dcddd47e255ef78b061f81c91a6475797dddb1e6ab7f23" exitCode=0 Oct 31 00:17:26.829949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:26.829895 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" event={"ID":"eff40f65-1fe2-42b7-8679-c6159b61c320","Type":"ContainerDied","Data":"7215369b1936776748dcddd47e255ef78b061f81c91a6475797dddb1e6ab7f23"} Oct 31 00:17:27.825848 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:27.825810 4969 patch_prober.go:29] interesting pod/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:17:27.826242 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:27.825863 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:17:28.028688 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.028666 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:28.071253 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.071216 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7"] Oct 31 00:17:28.075365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.075341 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-xks-ca\") pod \"eff40f65-1fe2-42b7-8679-c6159b61c320\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " Oct 31 00:17:28.075481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.075394 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff40f65-1fe2-42b7-8679-c6159b61c320-kube-api-access-2f78z\" (UniqueName: \"kubernetes.io/projected/eff40f65-1fe2-42b7-8679-c6159b61c320-kube-api-access-2f78z\") pod \"eff40f65-1fe2-42b7-8679-c6159b61c320\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " Oct 31 00:17:28.075481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.075451 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-openshift-ca\") pod \"eff40f65-1fe2-42b7-8679-c6159b61c320\" (UID: \"eff40f65-1fe2-42b7-8679-c6159b61c320\") " Oct 31 00:17:28.075914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.075859 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-openshift-ca" pod "eff40f65-1fe2-42b7-8679-c6159b61c320" (UID: "eff40f65-1fe2-42b7-8679-c6159b61c320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:28.103755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.103719 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-xks-ca" pod "eff40f65-1fe2-42b7-8679-c6159b61c320" (UID: "eff40f65-1fe2-42b7-8679-c6159b61c320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:28.103755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.103722 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff40f65-1fe2-42b7-8679-c6159b61c320-kube-api-access-2f78z" pod "eff40f65-1fe2-42b7-8679-c6159b61c320" (UID: "eff40f65-1fe2-42b7-8679-c6159b61c320"). InnerVolumeSpecName "kube-api-access-2f78z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:28.198619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.177126 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f78z\" (UniqueName: \"kubernetes.io/projected/eff40f65-1fe2-42b7-8679-c6159b61c320-kube-api-access-2f78z\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:28.198619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.177159 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:28.198619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.177177 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff40f65-1fe2-42b7-8679-c6159b61c320-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:28.825574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.825541 4969 patch_prober.go:29] interesting pod/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:17:28.825761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.825610 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:17:28.843546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.843517 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" Oct 31 00:17:28.843957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.843561 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp" event={"ID":"eff40f65-1fe2-42b7-8679-c6159b61c320","Type":"ContainerDied","Data":"ba1f47190d09be38da84c564f5b585ce3ee6b9aa8e577894bfed0bf645c18aca"} Oct 31 00:17:28.843957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:28.843633 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba1f47190d09be38da84c564f5b585ce3ee6b9aa8e577894bfed0bf645c18aca" Oct 31 00:17:29.825644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:29.825573 4969 patch_prober.go:29] interesting pod/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:17:29.825819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:29.825673 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:17:30.830085 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:30.830056 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:17:30.830518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:30.830500 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:17:30.831212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:30.831194 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:17:32.727193 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:17:32.727161 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:17:33.072265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.072179 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" podUID="dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb" containerName="payload-pre-processing" containerID="cri-o://f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9" gracePeriod=25 Oct 31 00:17:33.333996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.333943 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:17:33.372466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.372432 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-plugins-config-volume\") pod \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\" (UID: \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\") " Oct 31 00:17:33.372613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.372498 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-kube-api-access-4g9fx\" (UniqueName: \"kubernetes.io/projected/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-kube-api-access-4g9fx\") pod \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\" (UID: \"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb\") " Oct 31 00:17:33.372826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.372802 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-plugins-config-volume" pod "dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb" (UID: "dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:33.373016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.372996 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:33.374869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.374838 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-kube-api-access-4g9fx" pod "dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb" (UID: "dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb"). InnerVolumeSpecName "kube-api-access-4g9fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:33.474276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.474249 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g9fx\" (UniqueName: \"kubernetes.io/projected/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb-kube-api-access-4g9fx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:33.878791 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.878749 4969 generic.go:356] "Generic (PLEG): container finished" podID="dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb" containerID="f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9" exitCode=0 Oct 31 00:17:33.879196 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.878800 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" Oct 31 00:17:33.879196 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.878817 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" event={"ID":"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb","Type":"ContainerDied","Data":"f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9"} Oct 31 00:17:33.879196 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.878876 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7" event={"ID":"dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb","Type":"ContainerDied","Data":"b20f855d68e7cd45b8bee5fb5f3e6ad85032f7cc48712cb8ba5adc1d675bdee5"} Oct 31 00:17:33.879196 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.878907 4969 scope.go:122] "RemoveContainer" containerID="f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9" Oct 31 00:17:33.888878 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.888856 4969 scope.go:122] "RemoveContainer" containerID="f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9" Oct 31 00:17:33.889166 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:17:33.889141 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9\": container with ID starting with f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9 not found: ID does not exist" containerID="f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9" Oct 31 00:17:33.889269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:33.889176 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9"} err="failed to get container status \"f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9\": rpc error: code = NotFound desc = could not find container \"f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9\": container with ID starting with f5556d411067232f4dc2704c447d6dbd99709ba7160a3693e56467afa7debaf9 not found: ID does not exist" Oct 31 00:17:34.045217 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:34.045145 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7"] Oct 31 00:17:34.097287 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:34.097249 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e1bbe4b-5b6477dd99-wn5d7"] Oct 31 00:17:35.250281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:35.250248 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz"] Oct 31 00:17:35.733109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:35.733079 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb" path="/var/lib/kubelet/pods/dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb/volumes" Oct 31 00:17:37.046024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.045986 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248"] Oct 31 00:17:37.046691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.046671 4969 cpu_manager.go:414] "containerMap: removing container" podUID="dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb" containerName="payload-pre-processing" Oct 31 00:17:37.046802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.046693 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb" containerName="payload-pre-processing" Oct 31 00:17:37.046802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.046727 4969 cpu_manager.go:414] "containerMap: removing container" podUID="eff40f65-1fe2-42b7-8679-c6159b61c320" containerName="revoke-keys" Oct 31 00:17:37.046802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.046735 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff40f65-1fe2-42b7-8679-c6159b61c320" containerName="revoke-keys" Oct 31 00:17:37.046955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.046941 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad60ac5-4a7f-4cc9-9633-79bcbf65ecfb" containerName="payload-pre-processing" Oct 31 00:17:37.047006 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.046957 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff40f65-1fe2-42b7-8679-c6159b61c320" containerName="revoke-keys" Oct 31 00:17:37.051803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.051778 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.108073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.108035 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248"] Oct 31 00:17:37.119164 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.119117 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15eab294-vw248\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.119403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.119381 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15eab294-vw248\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.119502 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.119482 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvwn\" (UniqueName: \"kubernetes.io/projected/30090723-f6de-41ab-a0d8-2b1d8cd294c8-kube-api-access-njvwn\") pod \"maas-api-revoke-keys-e2e-ait-15eab294-vw248\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.125776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.125745 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:17:37.125776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.125751 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:17:37.221003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.220972 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15eab294-vw248\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.221170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.221026 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvwn\" (UniqueName: \"kubernetes.io/projected/30090723-f6de-41ab-a0d8-2b1d8cd294c8-kube-api-access-njvwn\") pod \"maas-api-revoke-keys-e2e-ait-15eab294-vw248\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.221170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.221051 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15eab294-vw248\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.221795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.221768 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15eab294-vw248\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.223516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.223497 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15eab294-vw248\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.282078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.282037 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvwn\" (UniqueName: \"kubernetes.io/projected/30090723-f6de-41ab-a0d8-2b1d8cd294c8-kube-api-access-njvwn\") pod \"maas-api-revoke-keys-e2e-ait-15eab294-vw248\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.369064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.369037 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:37.748437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.748411 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248"] Oct 31 00:17:37.749276 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:17:37.749251 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30090723_f6de_41ab_a0d8_2b1d8cd294c8.slice/crio-133a7536d42ebdca78ad4d8c3106e4c4bf3f87b274f5852e51360180d0768f6a WatchSource:0}: Error finding container 133a7536d42ebdca78ad4d8c3106e4c4bf3f87b274f5852e51360180d0768f6a: Status 404 returned error can't find the container with id 133a7536d42ebdca78ad4d8c3106e4c4bf3f87b274f5852e51360180d0768f6a Oct 31 00:17:37.929567 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:37.929534 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" event={"ID":"30090723-f6de-41ab-a0d8-2b1d8cd294c8","Type":"ContainerStarted","Data":"133a7536d42ebdca78ad4d8c3106e4c4bf3f87b274f5852e51360180d0768f6a"} Oct 31 00:17:38.937428 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:38.937394 4969 generic.go:356] "Generic (PLEG): container finished" podID="30090723-f6de-41ab-a0d8-2b1d8cd294c8" containerID="0b2a4e7acedce1f687d818695dd71a4ccfdc28e207986b492fcaad8f4ad8521a" exitCode=0 Oct 31 00:17:38.937844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:38.937442 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" event={"ID":"30090723-f6de-41ab-a0d8-2b1d8cd294c8","Type":"ContainerDied","Data":"0b2a4e7acedce1f687d818695dd71a4ccfdc28e207986b492fcaad8f4ad8521a"} Oct 31 00:17:39.665379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:39.665335 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd"] Oct 31 00:17:39.714377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:39.714341 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b78c516-nv4vd"] Oct 31 00:17:39.717530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:39.717506 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2"] Oct 31 00:17:39.717773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:39.717729 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" containerID="cri-o://6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2" gracePeriod=30 Oct 31 00:17:39.733810 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:39.733782 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab19133-b61c-41ee-84c4-3830e2f1bda1" path="/var/lib/kubelet/pods/bab19133-b61c-41ee-84c4-3830e2f1bda1/volumes" Oct 31 00:17:39.813364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:39.813332 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr"] Oct 31 00:17:39.813621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:39.813567 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" containerID="cri-o://73d49eb4839d17bea94e49f947fe5e12765336693b65336382e3d8aff0068cdf" gracePeriod=30 Oct 31 00:17:40.088296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.088268 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:40.148681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.148652 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-xks-ca\") pod \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " Oct 31 00:17:40.148825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.148702 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-openshift-ca\") pod \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " Oct 31 00:17:40.148825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.148780 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30090723-f6de-41ab-a0d8-2b1d8cd294c8-kube-api-access-njvwn\" (UniqueName: \"kubernetes.io/projected/30090723-f6de-41ab-a0d8-2b1d8cd294c8-kube-api-access-njvwn\") pod \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\" (UID: \"30090723-f6de-41ab-a0d8-2b1d8cd294c8\") " Oct 31 00:17:40.149082 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.149056 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-openshift-ca" pod "30090723-f6de-41ab-a0d8-2b1d8cd294c8" (UID: "30090723-f6de-41ab-a0d8-2b1d8cd294c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:40.149193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.149138 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:40.150834 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.150809 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-xks-ca" pod "30090723-f6de-41ab-a0d8-2b1d8cd294c8" (UID: "30090723-f6de-41ab-a0d8-2b1d8cd294c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:17:40.150946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.150932 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30090723-f6de-41ab-a0d8-2b1d8cd294c8-kube-api-access-njvwn" pod "30090723-f6de-41ab-a0d8-2b1d8cd294c8" (UID: "30090723-f6de-41ab-a0d8-2b1d8cd294c8"). InnerVolumeSpecName "kube-api-access-njvwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:40.250743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.250676 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30090723-f6de-41ab-a0d8-2b1d8cd294c8-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:40.250743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.250702 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njvwn\" (UniqueName: \"kubernetes.io/projected/30090723-f6de-41ab-a0d8-2b1d8cd294c8-kube-api-access-njvwn\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:40.250743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.250735 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" podUID="486a00d1-f042-4238-9b34-52552778e7ac" containerName="payload-processing" containerID="cri-o://ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217" gracePeriod=25 Oct 31 00:17:40.508340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.508253 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:17:40.553225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.553192 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486a00d1-f042-4238-9b34-52552778e7ac-kube-api-access-mpbp8\" (UniqueName: \"kubernetes.io/projected/486a00d1-f042-4238-9b34-52552778e7ac-kube-api-access-mpbp8\") pod \"486a00d1-f042-4238-9b34-52552778e7ac\" (UID: \"486a00d1-f042-4238-9b34-52552778e7ac\") " Oct 31 00:17:40.553421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.553294 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486a00d1-f042-4238-9b34-52552778e7ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486a00d1-f042-4238-9b34-52552778e7ac-plugins-config-volume\") pod \"486a00d1-f042-4238-9b34-52552778e7ac\" (UID: \"486a00d1-f042-4238-9b34-52552778e7ac\") " Oct 31 00:17:40.553728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.553703 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486a00d1-f042-4238-9b34-52552778e7ac-plugins-config-volume" pod "486a00d1-f042-4238-9b34-52552778e7ac" (UID: "486a00d1-f042-4238-9b34-52552778e7ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:40.555629 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.555579 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486a00d1-f042-4238-9b34-52552778e7ac-kube-api-access-mpbp8" pod "486a00d1-f042-4238-9b34-52552778e7ac" (UID: "486a00d1-f042-4238-9b34-52552778e7ac"). InnerVolumeSpecName "kube-api-access-mpbp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:40.654349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.654316 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpbp8\" (UniqueName: \"kubernetes.io/projected/486a00d1-f042-4238-9b34-52552778e7ac-kube-api-access-mpbp8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:40.654349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.654344 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486a00d1-f042-4238-9b34-52552778e7ac-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:40.724999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.724964 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp"] Oct 31 00:17:40.727583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.727562 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ab60e97-v6hbp"] Oct 31 00:17:40.951708 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.951675 4969 generic.go:356] "Generic (PLEG): container finished" podID="486a00d1-f042-4238-9b34-52552778e7ac" containerID="ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217" exitCode=0 Oct 31 00:17:40.951890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.951717 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" Oct 31 00:17:40.951890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.951720 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" event={"ID":"486a00d1-f042-4238-9b34-52552778e7ac","Type":"ContainerDied","Data":"ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217"} Oct 31 00:17:40.951890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.951779 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz" event={"ID":"486a00d1-f042-4238-9b34-52552778e7ac","Type":"ContainerDied","Data":"70f77b6fb2930ed123d96b709c32c23e4136b288d7d94682cedd1c4a9e07fa2b"} Oct 31 00:17:40.951890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.951803 4969 scope.go:122] "RemoveContainer" containerID="ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217" Oct 31 00:17:40.955830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.955809 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" event={"ID":"30090723-f6de-41ab-a0d8-2b1d8cd294c8","Type":"ContainerDied","Data":"133a7536d42ebdca78ad4d8c3106e4c4bf3f87b274f5852e51360180d0768f6a"} Oct 31 00:17:40.955954 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.955838 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248" Oct 31 00:17:40.955954 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.955846 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="133a7536d42ebdca78ad4d8c3106e4c4bf3f87b274f5852e51360180d0768f6a" Oct 31 00:17:40.964142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.964125 4969 scope.go:122] "RemoveContainer" containerID="ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217" Oct 31 00:17:40.964416 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:17:40.964399 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217\": container with ID starting with ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217 not found: ID does not exist" containerID="ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217" Oct 31 00:17:40.964473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.964423 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217"} err="failed to get container status \"ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217\": rpc error: code = NotFound desc = could not find container \"ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217\": container with ID starting with ad3573028bded24c04fe76142bb4c4ac0b4db74b789730f8d7974855726c2217 not found: ID does not exist" Oct 31 00:17:40.977822 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.977794 4969 patch_prober.go:29] interesting pod/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:17:40.977922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:40.977843 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:17:41.151561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.151527 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs"] Oct 31 00:17:41.152013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.151999 4969 cpu_manager.go:414] "containerMap: removing container" podUID="486a00d1-f042-4238-9b34-52552778e7ac" containerName="payload-processing" Oct 31 00:17:41.152056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.152015 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486a00d1-f042-4238-9b34-52552778e7ac" containerName="payload-processing" Oct 31 00:17:41.152056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.152037 4969 cpu_manager.go:414] "containerMap: removing container" podUID="30090723-f6de-41ab-a0d8-2b1d8cd294c8" containerName="revoke-keys" Oct 31 00:17:41.152056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.152042 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30090723-f6de-41ab-a0d8-2b1d8cd294c8" containerName="revoke-keys" Oct 31 00:17:41.152153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.152114 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="486a00d1-f042-4238-9b34-52552778e7ac" containerName="payload-processing" Oct 31 00:17:41.152153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.152124 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="30090723-f6de-41ab-a0d8-2b1d8cd294c8" containerName="revoke-keys" Oct 31 00:17:41.156618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.156581 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.180078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.180055 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz"] Oct 31 00:17:41.228279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.224870 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0144bd-openshift-default-dockercfg-4hs4v\"" Oct 31 00:17:41.229974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.229944 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ab60e97-c6f6d7f6c-pjrtz"] Oct 31 00:17:41.233417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.233388 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs"] Oct 31 00:17:41.260820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.260793 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.260946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.260842 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmlh\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-kube-api-access-bfmlh\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.260946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.260914 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0668f1-3bb4-4cad-976e-9b97244f5622-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.261026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.260956 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.261026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.260989 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.261131 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.261088 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.261187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.261172 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.261221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.261205 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.261259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.261234 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.362776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.362742 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmlh\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-kube-api-access-bfmlh\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.362776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.362776 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0668f1-3bb4-4cad-976e-9b97244f5622-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.362960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.362796 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.362960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.362930 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.362985 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.363073 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.363103 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.363118 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.363130 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.363212 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363458 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.363371 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.363496 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.363512 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0668f1-3bb4-4cad-976e-9b97244f5622-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.363561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.363534 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.365444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.365419 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.365548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.365533 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.426408 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.426378 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.426537 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.426391 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmlh\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-kube-api-access-bfmlh\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.469118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.469088 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:41.734505 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.734434 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486a00d1-f042-4238-9b34-52552778e7ac" path="/var/lib/kubelet/pods/486a00d1-f042-4238-9b34-52552778e7ac/volumes" Oct 31 00:17:41.734916 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.734900 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff40f65-1fe2-42b7-8679-c6159b61c320" path="/var/lib/kubelet/pods/eff40f65-1fe2-42b7-8679-c6159b61c320/volumes" Oct 31 00:17:41.907138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.907116 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs"] Oct 31 00:17:41.908755 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:17:41.908729 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0668f1_3bb4_4cad_976e_9b97244f5622.slice/crio-a75f139102b79b6fb092c333d9266b37f56c7c9b0ecaf4a66377bf6f7f98da16 WatchSource:0}: Error finding container a75f139102b79b6fb092c333d9266b37f56c7c9b0ecaf4a66377bf6f7f98da16: Status 404 returned error can't find the container with id a75f139102b79b6fb092c333d9266b37f56c7c9b0ecaf4a66377bf6f7f98da16 Oct 31 00:17:41.911230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.911197 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:41.911311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.911264 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:41.911311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.911294 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:17:41.967343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:41.967309 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" event={"ID":"6c0668f1-3bb4-4cad-976e-9b97244f5622","Type":"ContainerStarted","Data":"a75f139102b79b6fb092c333d9266b37f56c7c9b0ecaf4a66377bf6f7f98da16"} Oct 31 00:17:42.976392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:42.976356 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" event={"ID":"6c0668f1-3bb4-4cad-976e-9b97244f5622","Type":"ContainerStarted","Data":"8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708"} Oct 31 00:17:43.155624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:43.155561 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" podStartSLOduration=2.155546903 podStartE2EDuration="2.155546903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:43.152392543 +0000 UTC m=+895.627750966" watchObservedRunningTime="2026-10-31 00:17:43.155546903 +0000 UTC m=+895.630905388" Oct 31 00:17:43.470300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:43.470240 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:43.477979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:43.477952 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:43.982128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:43.982101 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:43.983001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:43.982985 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:17:44.985779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:44.985755 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:17:44.989820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:44.989766 4969 generic.go:356] "Generic (PLEG): container finished" podID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerID="73d49eb4839d17bea94e49f947fe5e12765336693b65336382e3d8aff0068cdf" exitCode=0 Oct 31 00:17:44.989934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:44.989839 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" event={"ID":"3d3ee639-6d22-4ce1-907a-0cc21f393eb5","Type":"ContainerDied","Data":"73d49eb4839d17bea94e49f947fe5e12765336693b65336382e3d8aff0068cdf"} Oct 31 00:17:44.989934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:44.989915 4969 scope.go:122] "RemoveContainer" containerID="9f66d3bc62f49f227f6a675daa02e7e3d13d31e89ff5bcc68a0262a8c491590a" Oct 31 00:17:44.991922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:44.991901 4969 generic.go:356] "Generic (PLEG): container finished" podID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerID="6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2" exitCode=0 Oct 31 00:17:44.992045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:44.991953 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" Oct 31 00:17:44.992045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:44.991973 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" event={"ID":"2a176f17-fe9b-4996-b2e5-cb715d4c5bec","Type":"ContainerDied","Data":"6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2"} Oct 31 00:17:44.992045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:44.992019 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2" event={"ID":"2a176f17-fe9b-4996-b2e5-cb715d4c5bec","Type":"ContainerDied","Data":"bd51220089c64823929f4f92db4ec97de92f54806100ab9be8e5889de54c774a"} Oct 31 00:17:45.006300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.006271 4969 scope.go:122] "RemoveContainer" containerID="6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2" Oct 31 00:17:45.015998 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.015975 4969 scope.go:122] "RemoveContainer" containerID="1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160" Oct 31 00:17:45.025888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.025810 4969 scope.go:122] "RemoveContainer" containerID="6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2" Oct 31 00:17:45.026137 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:17:45.026116 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2\": container with ID starting with 6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2 not found: ID does not exist" containerID="6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2" Oct 31 00:17:45.026229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.026152 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2"} err="failed to get container status \"6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2\": rpc error: code = NotFound desc = could not find container \"6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2\": container with ID starting with 6d7fb2e082475c8f55eaf45570e2e7a257ca0bcaca5672ccad602d6cd4b7dec2 not found: ID does not exist" Oct 31 00:17:45.026229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.026180 4969 scope.go:122] "RemoveContainer" containerID="1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160" Oct 31 00:17:45.026508 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:17:45.026487 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160\": container with ID starting with 1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160 not found: ID does not exist" containerID="1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160" Oct 31 00:17:45.026566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.026519 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160"} err="failed to get container status \"1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160\": rpc error: code = NotFound desc = could not find container \"1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160\": container with ID starting with 1673019273fb7a0a955661b52ca53d3456381110f14ba93a83c4d11808aa3160 not found: ID does not exist" Oct 31 00:17:45.079145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.079123 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:17:45.109122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109093 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istiod-ca-cert\") pod \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " Oct 31 00:17:45.109240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109139 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-podinfo\") pod \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " Oct 31 00:17:45.109240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109168 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-certs\") pod \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " Oct 31 00:17:45.109240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109206 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-credential-socket\") pod \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " Oct 31 00:17:45.109391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109248 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-token\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-token\") pod \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " Oct 31 00:17:45.109391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109281 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-kube-api-access-wz5vt\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-kube-api-access-wz5vt\") pod \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " Oct 31 00:17:45.109391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109314 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-data\") pod \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " Oct 31 00:17:45.109391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109364 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-socket\") pod \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " Oct 31 00:17:45.109618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109503 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-envoy\") pod \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\" (UID: \"2a176f17-fe9b-4996-b2e5-cb715d4c5bec\") " Oct 31 00:17:45.109618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109551 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-credential-socket" pod "2a176f17-fe9b-4996-b2e5-cb715d4c5bec" (UID: "2a176f17-fe9b-4996-b2e5-cb715d4c5bec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.109618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109566 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-certs" pod "2a176f17-fe9b-4996-b2e5-cb715d4c5bec" (UID: "2a176f17-fe9b-4996-b2e5-cb715d4c5bec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.109764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109623 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istiod-ca-cert" pod "2a176f17-fe9b-4996-b2e5-cb715d4c5bec" (UID: "2a176f17-fe9b-4996-b2e5-cb715d4c5bec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:45.109838 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109817 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-socket" pod "2a176f17-fe9b-4996-b2e5-cb715d4c5bec" (UID: "2a176f17-fe9b-4996-b2e5-cb715d4c5bec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.109927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.109903 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-data" pod "2a176f17-fe9b-4996-b2e5-cb715d4c5bec" (UID: "2a176f17-fe9b-4996-b2e5-cb715d4c5bec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.110370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.110344 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.110485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.110372 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.110485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.110388 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.110485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.110402 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.110485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.110416 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.111568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.111546 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-podinfo" pod "2a176f17-fe9b-4996-b2e5-cb715d4c5bec" (UID: "2a176f17-fe9b-4996-b2e5-cb715d4c5bec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:17:45.111698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.111647 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-token" pod "2a176f17-fe9b-4996-b2e5-cb715d4c5bec" (UID: "2a176f17-fe9b-4996-b2e5-cb715d4c5bec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:45.111763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.111721 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-kube-api-access-wz5vt" pod "2a176f17-fe9b-4996-b2e5-cb715d4c5bec" (UID: "2a176f17-fe9b-4996-b2e5-cb715d4c5bec"). InnerVolumeSpecName "kube-api-access-wz5vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:45.111811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.111796 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-envoy" pod "2a176f17-fe9b-4996-b2e5-cb715d4c5bec" (UID: "2a176f17-fe9b-4996-b2e5-cb715d4c5bec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.211471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211445 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-data\") pod \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " Oct 31 00:17:45.211644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211483 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-token\") pod \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " Oct 31 00:17:45.211644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211517 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-socket\") pod \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " Oct 31 00:17:45.211644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211564 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-certs\") pod \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " Oct 31 00:17:45.211644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211587 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-kube-api-access-w5v9s\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-kube-api-access-w5v9s\") pod \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " Oct 31 00:17:45.211644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211626 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istiod-ca-cert\") pod \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " Oct 31 00:17:45.211919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211655 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-envoy\") pod \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " Oct 31 00:17:45.211919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211729 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-podinfo\") pod \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " Oct 31 00:17:45.211919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211741 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-data" pod "3d3ee639-6d22-4ce1-907a-0cc21f393eb5" (UID: "3d3ee639-6d22-4ce1-907a-0cc21f393eb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.211919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211766 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-credential-socket\") pod \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\" (UID: \"3d3ee639-6d22-4ce1-907a-0cc21f393eb5\") " Oct 31 00:17:45.211919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211850 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-socket" pod "3d3ee639-6d22-4ce1-907a-0cc21f393eb5" (UID: "3d3ee639-6d22-4ce1-907a-0cc21f393eb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.211919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211890 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-certs" pod "3d3ee639-6d22-4ce1-907a-0cc21f393eb5" (UID: "3d3ee639-6d22-4ce1-907a-0cc21f393eb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.212234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.211982 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-credential-socket" pod "3d3ee639-6d22-4ce1-907a-0cc21f393eb5" (UID: "3d3ee639-6d22-4ce1-907a-0cc21f393eb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.212234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.212226 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istiod-ca-cert" pod "3d3ee639-6d22-4ce1-907a-0cc21f393eb5" (UID: "3d3ee639-6d22-4ce1-907a-0cc21f393eb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:45.212340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.212252 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.212340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.212281 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz5vt\" (UniqueName: \"kubernetes.io/projected/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-kube-api-access-wz5vt\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.212340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.212297 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.212340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.212311 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.212340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.212326 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.212340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.212339 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.212849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.212353 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.212849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.212366 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a176f17-fe9b-4996-b2e5-cb715d4c5bec-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.214101 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.214079 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-kube-api-access-w5v9s" pod "3d3ee639-6d22-4ce1-907a-0cc21f393eb5" (UID: "3d3ee639-6d22-4ce1-907a-0cc21f393eb5"). InnerVolumeSpecName "kube-api-access-w5v9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:45.214199 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.214142 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-envoy" pod "3d3ee639-6d22-4ce1-907a-0cc21f393eb5" (UID: "3d3ee639-6d22-4ce1-907a-0cc21f393eb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:17:45.214238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.214206 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-podinfo" pod "3d3ee639-6d22-4ce1-907a-0cc21f393eb5" (UID: "3d3ee639-6d22-4ce1-907a-0cc21f393eb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:17:45.214479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.214450 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-token" pod "3d3ee639-6d22-4ce1-907a-0cc21f393eb5" (UID: "3d3ee639-6d22-4ce1-907a-0cc21f393eb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:45.313487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.313460 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.313487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.313487 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.313684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.313500 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5v9s\" (UniqueName: \"kubernetes.io/projected/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-kube-api-access-w5v9s\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.313684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.313514 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.313684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.313528 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3ee639-6d22-4ce1-907a-0cc21f393eb5-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:45.442297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.442266 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2"] Oct 31 00:17:45.454543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.454519 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0e1bbe4b-openshift-default-f46944f8c-ncwd2"] Oct 31 00:17:45.735546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:45.735515 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" path="/var/lib/kubelet/pods/2a176f17-fe9b-4996-b2e5-cb715d4c5bec/volumes" Oct 31 00:17:46.002515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:46.002422 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" event={"ID":"3d3ee639-6d22-4ce1-907a-0cc21f393eb5","Type":"ContainerDied","Data":"13c6dcbbeecd17a540eae0ca6a0d4d25021cee3d91b30b2d6ac06ba69b4086c5"} Oct 31 00:17:46.002515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:46.002444 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr" Oct 31 00:17:46.002515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:46.002485 4969 scope.go:122] "RemoveContainer" containerID="73d49eb4839d17bea94e49f947fe5e12765336693b65336382e3d8aff0068cdf" Oct 31 00:17:46.175890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:46.175859 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr"] Oct 31 00:17:46.181515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:46.181488 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4b78c516-gw-openshift-default-6dc7ccd9c-zmpzr"] Oct 31 00:17:47.733610 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:47.733559 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" path="/var/lib/kubelet/pods/3d3ee639-6d22-4ce1-907a-0cc21f393eb5/volumes" Oct 31 00:17:47.804902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:47.804874 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:17:47.805071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:47.805059 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:17:47.831658 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:47.831635 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:17:47.832170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:47.832142 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:17:49.383818 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.383783 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np"] Oct 31 00:17:49.384280 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384267 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" Oct 31 00:17:49.384322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384281 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" Oct 31 00:17:49.384322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384292 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" Oct 31 00:17:49.384322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384296 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" Oct 31 00:17:49.384322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384305 4969 cpu_manager.go:414] "containerMap: removing container" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" Oct 31 00:17:49.384322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384310 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" Oct 31 00:17:49.384322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384318 4969 cpu_manager.go:414] "containerMap: removing container" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" Oct 31 00:17:49.384322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384323 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" Oct 31 00:17:49.384572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384406 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" Oct 31 00:17:49.384572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384413 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a176f17-fe9b-4996-b2e5-cb715d4c5bec" containerName="istio-proxy" Oct 31 00:17:49.384572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384419 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" Oct 31 00:17:49.384572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.384426 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3ee639-6d22-4ce1-907a-0cc21f393eb5" containerName="istio-proxy" Oct 31 00:17:49.389408 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.389353 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:17:49.446307 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.446276 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np"] Oct 31 00:17:49.460262 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.460224 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3487e6ad-e165-46d6-942d-f9c25c800ba5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np\" (UID: \"3487e6ad-e165-46d6-942d-f9c25c800ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:17:49.460629 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.460328 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8p27\" (UniqueName: \"kubernetes.io/projected/3487e6ad-e165-46d6-942d-f9c25c800ba5-kube-api-access-p8p27\") pod \"payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np\" (UID: \"3487e6ad-e165-46d6-942d-f9c25c800ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:17:49.492573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.492540 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-7abb95\"" Oct 31 00:17:49.492771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.492623 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-7abb95-dockercfg-gpsf4\"" Oct 31 00:17:49.545426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.545386 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg"] Oct 31 00:17:49.549267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.549243 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:17:49.561219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.561192 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3487e6ad-e165-46d6-942d-f9c25c800ba5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np\" (UID: \"3487e6ad-e165-46d6-942d-f9c25c800ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:17:49.561310 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.561245 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8p27\" (UniqueName: \"kubernetes.io/projected/3487e6ad-e165-46d6-942d-f9c25c800ba5-kube-api-access-p8p27\") pod \"payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np\" (UID: \"3487e6ad-e165-46d6-942d-f9c25c800ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:17:49.561904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.561885 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3487e6ad-e165-46d6-942d-f9c25c800ba5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np\" (UID: \"3487e6ad-e165-46d6-942d-f9c25c800ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:17:49.590862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.590840 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg"] Oct 31 00:17:49.622460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.622436 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8p27\" (UniqueName: \"kubernetes.io/projected/3487e6ad-e165-46d6-942d-f9c25c800ba5-kube-api-access-p8p27\") pod \"payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np\" (UID: \"3487e6ad-e165-46d6-942d-f9c25c800ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:17:49.662550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.662481 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59809d8f-4894-49ce-ab6c-361e00e41285-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg\" (UID: \"59809d8f-4894-49ce-ab6c-361e00e41285\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:17:49.662702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.662559 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m2sf\" (UniqueName: \"kubernetes.io/projected/59809d8f-4894-49ce-ab6c-361e00e41285-kube-api-access-8m2sf\") pod \"payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg\" (UID: \"59809d8f-4894-49ce-ab6c-361e00e41285\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:17:49.702721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.702692 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:17:49.764180 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.764149 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59809d8f-4894-49ce-ab6c-361e00e41285-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg\" (UID: \"59809d8f-4894-49ce-ab6c-361e00e41285\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:17:49.764269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.764229 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m2sf\" (UniqueName: \"kubernetes.io/projected/59809d8f-4894-49ce-ab6c-361e00e41285-kube-api-access-8m2sf\") pod \"payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg\" (UID: \"59809d8f-4894-49ce-ab6c-361e00e41285\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:17:49.765233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.765210 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59809d8f-4894-49ce-ab6c-361e00e41285-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg\" (UID: \"59809d8f-4894-49ce-ab6c-361e00e41285\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:17:49.825804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.825771 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m2sf\" (UniqueName: \"kubernetes.io/projected/59809d8f-4894-49ce-ab6c-361e00e41285-kube-api-access-8m2sf\") pod \"payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg\" (UID: \"59809d8f-4894-49ce-ab6c-361e00e41285\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:17:49.862640 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:49.862615 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:17:50.079753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:50.079727 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np"] Oct 31 00:17:50.080364 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:17:50.080338 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3487e6ad_e165_46d6_942d_f9c25c800ba5.slice/crio-6e77b2815f6be9296bd9c67861d8e4e709d1884a193e83bc35a1853273360525 WatchSource:0}: Error finding container 6e77b2815f6be9296bd9c67861d8e4e709d1884a193e83bc35a1853273360525: Status 404 returned error can't find the container with id 6e77b2815f6be9296bd9c67861d8e4e709d1884a193e83bc35a1853273360525 Oct 31 00:17:50.088322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:50.088297 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49"] Oct 31 00:17:50.195086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:50.195015 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d"] Oct 31 00:17:50.237311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:50.237290 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg"] Oct 31 00:17:50.239410 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:17:50.239376 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59809d8f_4894_49ce_ab6c_361e00e41285.slice/crio-2e28d7f44fc38d54ab7846d45f22829708999f9ed98c8cbb8cccec7321ccae96 WatchSource:0}: Error finding container 2e28d7f44fc38d54ab7846d45f22829708999f9ed98c8cbb8cccec7321ccae96: Status 404 returned error can't find the container with id 2e28d7f44fc38d54ab7846d45f22829708999f9ed98c8cbb8cccec7321ccae96 Oct 31 00:17:51.035008 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:51.034969 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" event={"ID":"59809d8f-4894-49ce-ab6c-361e00e41285","Type":"ContainerStarted","Data":"dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7"} Oct 31 00:17:51.035512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:51.035054 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" event={"ID":"59809d8f-4894-49ce-ab6c-361e00e41285","Type":"ContainerStarted","Data":"2e28d7f44fc38d54ab7846d45f22829708999f9ed98c8cbb8cccec7321ccae96"} Oct 31 00:17:51.035512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:51.035078 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:17:51.036716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:51.036691 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" event={"ID":"3487e6ad-e165-46d6-942d-f9c25c800ba5","Type":"ContainerStarted","Data":"a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851"} Oct 31 00:17:51.036813 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:51.036729 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" event={"ID":"3487e6ad-e165-46d6-942d-f9c25c800ba5","Type":"ContainerStarted","Data":"6e77b2815f6be9296bd9c67861d8e4e709d1884a193e83bc35a1853273360525"} Oct 31 00:17:51.036862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:51.036825 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:17:51.155184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:51.155131 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" podStartSLOduration=2.155112989 podStartE2EDuration="2.155112989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:51.151491421 +0000 UTC m=+903.626849844" watchObservedRunningTime="2026-10-31 00:17:51.155112989 +0000 UTC m=+903.630471411" Oct 31 00:17:51.272280 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:51.272224 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" podStartSLOduration=2.272203922 podStartE2EDuration="2.272203922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:51.268305386 +0000 UTC m=+903.743663833" watchObservedRunningTime="2026-10-31 00:17:51.272203922 +0000 UTC m=+903.747562348" Oct 31 00:17:55.089325 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.089287 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" podUID="9d08a915-f585-4694-b8f7-447e8739abb7" containerName="payload-processing" containerID="cri-o://0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb" gracePeriod=25 Oct 31 00:17:55.195931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.195891 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" podUID="28e52a4d-c6f6-4b36-804b-b2911893dc2c" containerName="payload-pre-processing" containerID="cri-o://609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da" gracePeriod=25 Oct 31 00:17:55.351763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.351690 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:17:55.467735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.467708 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:17:55.520721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.520692 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d08a915-f585-4694-b8f7-447e8739abb7-kube-api-access-ddzfj\" (UniqueName: \"kubernetes.io/projected/9d08a915-f585-4694-b8f7-447e8739abb7-kube-api-access-ddzfj\") pod \"9d08a915-f585-4694-b8f7-447e8739abb7\" (UID: \"9d08a915-f585-4694-b8f7-447e8739abb7\") " Oct 31 00:17:55.520903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.520786 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d08a915-f585-4694-b8f7-447e8739abb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d08a915-f585-4694-b8f7-447e8739abb7-plugins-config-volume\") pod \"9d08a915-f585-4694-b8f7-447e8739abb7\" (UID: \"9d08a915-f585-4694-b8f7-447e8739abb7\") " Oct 31 00:17:55.521200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.521178 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d08a915-f585-4694-b8f7-447e8739abb7-plugins-config-volume" pod "9d08a915-f585-4694-b8f7-447e8739abb7" (UID: "9d08a915-f585-4694-b8f7-447e8739abb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:55.522974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.522953 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d08a915-f585-4694-b8f7-447e8739abb7-kube-api-access-ddzfj" pod "9d08a915-f585-4694-b8f7-447e8739abb7" (UID: "9d08a915-f585-4694-b8f7-447e8739abb7"). InnerVolumeSpecName "kube-api-access-ddzfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:55.621722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.621687 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e52a4d-c6f6-4b36-804b-b2911893dc2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e52a4d-c6f6-4b36-804b-b2911893dc2c-plugins-config-volume\") pod \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\" (UID: \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\") " Oct 31 00:17:55.621906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.621853 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e52a4d-c6f6-4b36-804b-b2911893dc2c-kube-api-access-qgkrt\" (UniqueName: \"kubernetes.io/projected/28e52a4d-c6f6-4b36-804b-b2911893dc2c-kube-api-access-qgkrt\") pod \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\" (UID: \"28e52a4d-c6f6-4b36-804b-b2911893dc2c\") " Oct 31 00:17:55.622060 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.622037 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e52a4d-c6f6-4b36-804b-b2911893dc2c-plugins-config-volume" pod "28e52a4d-c6f6-4b36-804b-b2911893dc2c" (UID: "28e52a4d-c6f6-4b36-804b-b2911893dc2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:17:55.622188 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.622171 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddzfj\" (UniqueName: \"kubernetes.io/projected/9d08a915-f585-4694-b8f7-447e8739abb7-kube-api-access-ddzfj\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:55.622248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.622196 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d08a915-f585-4694-b8f7-447e8739abb7-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:55.622248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.622214 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e52a4d-c6f6-4b36-804b-b2911893dc2c-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:55.624017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.623997 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e52a4d-c6f6-4b36-804b-b2911893dc2c-kube-api-access-qgkrt" pod "28e52a4d-c6f6-4b36-804b-b2911893dc2c" (UID: "28e52a4d-c6f6-4b36-804b-b2911893dc2c"). InnerVolumeSpecName "kube-api-access-qgkrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:17:55.730114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:55.723696 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgkrt\" (UniqueName: \"kubernetes.io/projected/28e52a4d-c6f6-4b36-804b-b2911893dc2c-kube-api-access-qgkrt\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:17:56.069616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.069511 4969 generic.go:356] "Generic (PLEG): container finished" podID="9d08a915-f585-4694-b8f7-447e8739abb7" containerID="0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb" exitCode=0 Oct 31 00:17:56.069616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.069578 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" Oct 31 00:17:56.069870 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.069620 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" event={"ID":"9d08a915-f585-4694-b8f7-447e8739abb7","Type":"ContainerDied","Data":"0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb"} Oct 31 00:17:56.069870 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.069686 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49" event={"ID":"9d08a915-f585-4694-b8f7-447e8739abb7","Type":"ContainerDied","Data":"fc50488de2b6358cb093da1d35daa6f5696ae417c23653fff96337bfface114a"} Oct 31 00:17:56.069870 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.069719 4969 scope.go:122] "RemoveContainer" containerID="0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb" Oct 31 00:17:56.071432 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.071410 4969 generic.go:356] "Generic (PLEG): container finished" podID="28e52a4d-c6f6-4b36-804b-b2911893dc2c" containerID="609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da" exitCode=0 Oct 31 00:17:56.071554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.071486 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" Oct 31 00:17:56.071673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.071570 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" event={"ID":"28e52a4d-c6f6-4b36-804b-b2911893dc2c","Type":"ContainerDied","Data":"609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da"} Oct 31 00:17:56.071673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.071654 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d" event={"ID":"28e52a4d-c6f6-4b36-804b-b2911893dc2c","Type":"ContainerDied","Data":"dc4159609fee3dc15117e987faa660ba7943bbde63b2577a7339aef3db193441"} Oct 31 00:17:56.080064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.080054 4969 scope.go:122] "RemoveContainer" containerID="0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb" Oct 31 00:17:56.080312 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:17:56.080296 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb\": container with ID starting with 0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb not found: ID does not exist" containerID="0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb" Oct 31 00:17:56.080358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.080319 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb"} err="failed to get container status \"0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb\": rpc error: code = NotFound desc = could not find container \"0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb\": container with ID starting with 0789eb48068dfcd55d27c469a28109513a39f839638d6fd506f9aaa99d35dfdb not found: ID does not exist" Oct 31 00:17:56.080358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.080336 4969 scope.go:122] "RemoveContainer" containerID="609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da" Oct 31 00:17:56.088490 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.088474 4969 scope.go:122] "RemoveContainer" containerID="609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da" Oct 31 00:17:56.088755 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:17:56.088737 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da\": container with ID starting with 609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da not found: ID does not exist" containerID="609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da" Oct 31 00:17:56.088833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.088763 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da"} err="failed to get container status \"609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da\": rpc error: code = NotFound desc = could not find container \"609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da\": container with ID starting with 609ad58c201562a87be864af7bbae2dc84e021ebf17e1d18ecb305c1a932f2da not found: ID does not exist" Oct 31 00:17:56.234555 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.234520 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d"] Oct 31 00:17:56.279404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.279374 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15eab294-5b4467ddbc-j7c4d"] Oct 31 00:17:56.347334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.347249 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49"] Oct 31 00:17:56.353462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.353426 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15eab294-6db845f85-gzp49"] Oct 31 00:17:56.667228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.667197 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd"] Oct 31 00:17:56.667719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.667701 4969 cpu_manager.go:414] "containerMap: removing container" podUID="28e52a4d-c6f6-4b36-804b-b2911893dc2c" containerName="payload-pre-processing" Oct 31 00:17:56.667840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.667721 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e52a4d-c6f6-4b36-804b-b2911893dc2c" containerName="payload-pre-processing" Oct 31 00:17:56.667840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.667751 4969 cpu_manager.go:414] "containerMap: removing container" podUID="9d08a915-f585-4694-b8f7-447e8739abb7" containerName="payload-processing" Oct 31 00:17:56.667840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.667759 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d08a915-f585-4694-b8f7-447e8739abb7" containerName="payload-processing" Oct 31 00:17:56.667994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.667911 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e52a4d-c6f6-4b36-804b-b2911893dc2c" containerName="payload-pre-processing" Oct 31 00:17:56.667994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.667927 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d08a915-f585-4694-b8f7-447e8739abb7" containerName="payload-processing" Oct 31 00:17:56.672644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.672622 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:17:56.725652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.725555 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0144bd\"" Oct 31 00:17:56.725809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.725729 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0144bd-dockercfg-hbxmq\"" Oct 31 00:17:56.729417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.729396 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd"] Oct 31 00:17:56.816404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.816368 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j"] Oct 31 00:17:56.820100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.820084 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:17:56.835267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.835243 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgnzj\" (UniqueName: \"kubernetes.io/projected/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-kube-api-access-pgnzj\") pod \"payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd\" (UID: \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:17:56.835423 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.835401 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd\" (UID: \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:17:56.878224 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.878193 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j"] Oct 31 00:17:56.936176 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.936091 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9sfb\" (UniqueName: \"kubernetes.io/projected/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-kube-api-access-s9sfb\") pod \"payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j\" (UID: \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:17:56.936176 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.936126 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j\" (UID: \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:17:56.936387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.936308 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd\" (UID: \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:17:56.936470 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.936452 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgnzj\" (UniqueName: \"kubernetes.io/projected/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-kube-api-access-pgnzj\") pod \"payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd\" (UID: \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:17:56.937218 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.937195 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd\" (UID: \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:17:56.998009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:56.997986 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgnzj\" (UniqueName: \"kubernetes.io/projected/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-kube-api-access-pgnzj\") pod \"payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd\" (UID: \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:17:57.037248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.037222 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9sfb\" (UniqueName: \"kubernetes.io/projected/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-kube-api-access-s9sfb\") pod \"payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j\" (UID: \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:17:57.037342 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.037253 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j\" (UID: \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:17:57.037939 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.037921 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j\" (UID: \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:17:57.098539 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.098512 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9sfb\" (UniqueName: \"kubernetes.io/projected/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-kube-api-access-s9sfb\") pod \"payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j\" (UID: \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:17:57.133139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.133109 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:17:57.286126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.286095 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:17:57.510408 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.510174 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j"] Oct 31 00:17:57.513996 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:17:57.513964 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2d7e43_d4ba_4d8e_8a37_80b52b5e3bc4.slice/crio-e8b40ca33473e44dee10896ec70ec90d378fe51708be56a40c74ad6d751e30e4 WatchSource:0}: Error finding container e8b40ca33473e44dee10896ec70ec90d378fe51708be56a40c74ad6d751e30e4: Status 404 returned error can't find the container with id e8b40ca33473e44dee10896ec70ec90d378fe51708be56a40c74ad6d751e30e4 Oct 31 00:17:57.663947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.663924 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd"] Oct 31 00:17:57.664306 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:17:57.664287 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890b0974_c3c4_4d27_a1b8_fb3f018b14dc.slice/crio-3f6221a0ebccbc2ed74b0412e9b1c8cc5da5f516ed942e42b0b3c52f580b823c WatchSource:0}: Error finding container 3f6221a0ebccbc2ed74b0412e9b1c8cc5da5f516ed942e42b0b3c52f580b823c: Status 404 returned error can't find the container with id 3f6221a0ebccbc2ed74b0412e9b1c8cc5da5f516ed942e42b0b3c52f580b823c Oct 31 00:17:57.740932 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.740895 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e52a4d-c6f6-4b36-804b-b2911893dc2c" path="/var/lib/kubelet/pods/28e52a4d-c6f6-4b36-804b-b2911893dc2c/volumes" Oct 31 00:17:57.741283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:57.741270 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d08a915-f585-4694-b8f7-447e8739abb7" path="/var/lib/kubelet/pods/9d08a915-f585-4694-b8f7-447e8739abb7/volumes" Oct 31 00:17:58.087669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:58.087630 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" event={"ID":"890b0974-c3c4-4d27-a1b8-fb3f018b14dc","Type":"ContainerStarted","Data":"953ead5d0d4f2ddbec71025ca8907c89aab07b6bbaeceb30e6e17d393cc5bf40"} Oct 31 00:17:58.087867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:58.087691 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" event={"ID":"890b0974-c3c4-4d27-a1b8-fb3f018b14dc","Type":"ContainerStarted","Data":"3f6221a0ebccbc2ed74b0412e9b1c8cc5da5f516ed942e42b0b3c52f580b823c"} Oct 31 00:17:58.087867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:58.087773 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:17:58.089363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:58.089328 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" event={"ID":"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4","Type":"ContainerStarted","Data":"db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735"} Oct 31 00:17:58.089491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:58.089374 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" event={"ID":"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4","Type":"ContainerStarted","Data":"e8b40ca33473e44dee10896ec70ec90d378fe51708be56a40c74ad6d751e30e4"} Oct 31 00:17:58.089491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:58.089402 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:17:58.204291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:58.204239 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" podStartSLOduration=2.204225721 podStartE2EDuration="2.204225721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:58.203399907 +0000 UTC m=+910.678758330" watchObservedRunningTime="2026-10-31 00:17:58.204225721 +0000 UTC m=+910.679584143" Oct 31 00:17:58.321252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:17:58.321203 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" podStartSLOduration=2.32118763 podStartE2EDuration="2.32118763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:58.319363801 +0000 UTC m=+910.794722219" watchObservedRunningTime="2026-10-31 00:17:58.32118763 +0000 UTC m=+910.796546100" Oct 31 00:18:01.677934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.677895 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8"] Oct 31 00:18:01.682869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.682850 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:01.789486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.789423 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-10a7aa\"" Oct 31 00:18:01.789872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.789833 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-10a7aa-dockercfg-mvgzg\"" Oct 31 00:18:01.790108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.790067 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7gq\" (UniqueName: \"kubernetes.io/projected/cd9eb718-1650-4ba0-9a5c-362cffb9354e-kube-api-access-ll7gq\") pod \"payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8\" (UID: \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:01.790271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.790240 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb718-1650-4ba0-9a5c-362cffb9354e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8\" (UID: \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:01.794992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.794965 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8"] Oct 31 00:18:01.842022 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.841985 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h"] Oct 31 00:18:01.845905 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.845886 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:01.890939 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.890896 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7gq\" (UniqueName: \"kubernetes.io/projected/cd9eb718-1650-4ba0-9a5c-362cffb9354e-kube-api-access-ll7gq\") pod \"payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8\" (UID: \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:01.891120 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.891022 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb718-1650-4ba0-9a5c-362cffb9354e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8\" (UID: \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:01.891699 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.891679 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb718-1650-4ba0-9a5c-362cffb9354e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8\" (UID: \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:01.901904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.901872 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h"] Oct 31 00:18:01.952371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.952301 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7gq\" (UniqueName: \"kubernetes.io/projected/cd9eb718-1650-4ba0-9a5c-362cffb9354e-kube-api-access-ll7gq\") pod \"payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8\" (UID: \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:01.992540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.992502 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7b692f-c20b-4754-ad19-c0fad273c3cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h\" (UID: \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:01.992714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.992575 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25pbh\" (UniqueName: \"kubernetes.io/projected/1c7b692f-c20b-4754-ad19-c0fad273c3cf-kube-api-access-25pbh\") pod \"payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h\" (UID: \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:01.996425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:01.996403 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:02.045674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:02.045644 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 00:18:02.045861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:02.045834 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 00:18:02.093514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:02.093479 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25pbh\" (UniqueName: \"kubernetes.io/projected/1c7b692f-c20b-4754-ad19-c0fad273c3cf-kube-api-access-25pbh\") pod \"payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h\" (UID: \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:02.093709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:02.093621 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7b692f-c20b-4754-ad19-c0fad273c3cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h\" (UID: \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:02.094375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:02.094349 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7b692f-c20b-4754-ad19-c0fad273c3cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h\" (UID: \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:02.157127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:02.157097 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25pbh\" (UniqueName: \"kubernetes.io/projected/1c7b692f-c20b-4754-ad19-c0fad273c3cf-kube-api-access-25pbh\") pod \"payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h\" (UID: \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:02.160492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:02.160476 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:02.374843 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:02.374821 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8"] Oct 31 00:18:02.376032 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:18:02.376004 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9eb718_1650_4ba0_9a5c_362cffb9354e.slice/crio-82c02f16594ab49d9a40959cf54307e72891ddb3918e6f208fcd1a13bc8f791c WatchSource:0}: Error finding container 82c02f16594ab49d9a40959cf54307e72891ddb3918e6f208fcd1a13bc8f791c: Status 404 returned error can't find the container with id 82c02f16594ab49d9a40959cf54307e72891ddb3918e6f208fcd1a13bc8f791c Oct 31 00:18:02.552879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:02.552853 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h"] Oct 31 00:18:02.553270 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:18:02.553247 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7b692f_c20b_4754_ad19_c0fad273c3cf.slice/crio-94760dd9a7a0ab5fb93fe6f27a34bedd0446ea06ddf043a75c554af55c58e33c WatchSource:0}: Error finding container 94760dd9a7a0ab5fb93fe6f27a34bedd0446ea06ddf043a75c554af55c58e33c: Status 404 returned error can't find the container with id 94760dd9a7a0ab5fb93fe6f27a34bedd0446ea06ddf043a75c554af55c58e33c Oct 31 00:18:02.727044 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:18:02.727009 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:18:03.126173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:03.126131 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" event={"ID":"cd9eb718-1650-4ba0-9a5c-362cffb9354e","Type":"ContainerStarted","Data":"38b711041bf7efdaf04d6ff556465b62263182f5e65f8f3c82fb5c8f658a4174"} Oct 31 00:18:03.126348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:03.126195 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:03.126348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:03.126214 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" event={"ID":"cd9eb718-1650-4ba0-9a5c-362cffb9354e","Type":"ContainerStarted","Data":"82c02f16594ab49d9a40959cf54307e72891ddb3918e6f208fcd1a13bc8f791c"} Oct 31 00:18:03.127875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:03.127847 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" event={"ID":"1c7b692f-c20b-4754-ad19-c0fad273c3cf","Type":"ContainerStarted","Data":"de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a"} Oct 31 00:18:03.127976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:03.127899 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" event={"ID":"1c7b692f-c20b-4754-ad19-c0fad273c3cf","Type":"ContainerStarted","Data":"94760dd9a7a0ab5fb93fe6f27a34bedd0446ea06ddf043a75c554af55c58e33c"} Oct 31 00:18:03.127976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:03.127928 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:03.242578 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:03.242524 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" podStartSLOduration=2.242509961 podStartE2EDuration="2.242509961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:03.240435238 +0000 UTC m=+915.715793662" watchObservedRunningTime="2026-10-31 00:18:03.242509961 +0000 UTC m=+915.717868402" Oct 31 00:18:03.358585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:03.358528 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" podStartSLOduration=2.358508789 podStartE2EDuration="2.358508789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:03.355296025 +0000 UTC m=+915.830654448" watchObservedRunningTime="2026-10-31 00:18:03.358508789 +0000 UTC m=+915.833867216" Oct 31 00:18:09.098391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:09.098366 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 00:18:09.098858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:09.098443 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 00:18:11.959873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:11.959836 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248"] Oct 31 00:18:12.009236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:12.009203 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15eab294-vw248"] Oct 31 00:18:12.041216 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:12.041185 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9"] Oct 31 00:18:12.041454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:12.041411 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" containerID="cri-o://978465d2019fa539b8ae78c61a671d84a4f6c9e0271965643d4a690c333ae113" gracePeriod=30 Oct 31 00:18:13.446342 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.446297 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj"] Oct 31 00:18:13.450565 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.450540 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.494087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.494046 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.494263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.494218 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.494263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.494246 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.494352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.494286 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.494352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.494320 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf8r6\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-kube-api-access-gf8r6\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.494425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.494366 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.494425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.494397 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.494499 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.494427 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.494499 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.494490 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.503637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.502830 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e1ca71-openshift-default-dockercfg-sh7lb\"" Oct 31 00:18:13.556023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.555989 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj"] Oct 31 00:18:13.595893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.595860 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.595909 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.595950 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.595983 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf8r6\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-kube-api-access-gf8r6\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596029 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596058 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596084 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596171 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596213 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596379 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596408 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596579 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596662 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.596901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.596881 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.598275 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.598255 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.598541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.598510 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.657236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.657207 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.657416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.657354 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf8r6\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-kube-api-access-gf8r6\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:13.733283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.733213 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30090723-f6de-41ab-a0d8-2b1d8cd294c8" path="/var/lib/kubelet/pods/30090723-f6de-41ab-a0d8-2b1d8cd294c8/volumes" Oct 31 00:18:13.764318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:13.764288 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:14.137083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:14.137055 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 00:18:14.137275 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:14.137127 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 00:18:14.200521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:14.200496 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj"] Oct 31 00:18:14.201752 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:18:14.201725 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32982e9_15ef_45ac_ab4c_55f742b5bc4c.slice/crio-c398eadd2d44d3da1977a4d84e045d7495f585957e08db4aa492d2b074d6539f WatchSource:0}: Error finding container c398eadd2d44d3da1977a4d84e045d7495f585957e08db4aa492d2b074d6539f: Status 404 returned error can't find the container with id c398eadd2d44d3da1977a4d84e045d7495f585957e08db4aa492d2b074d6539f Oct 31 00:18:14.204067 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:14.204036 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:14.204154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:14.204096 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:14.204154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:14.204127 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:15.212694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:15.212652 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" event={"ID":"f32982e9-15ef-45ac-ab4c-55f742b5bc4c","Type":"ContainerStarted","Data":"768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b"} Oct 31 00:18:15.213140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:15.212713 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" event={"ID":"f32982e9-15ef-45ac-ab4c-55f742b5bc4c","Type":"ContainerStarted","Data":"c398eadd2d44d3da1977a4d84e045d7495f585957e08db4aa492d2b074d6539f"} Oct 31 00:18:15.438607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:15.438555 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" podStartSLOduration=2.438540787 podStartE2EDuration="2.438540787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:15.387006601 +0000 UTC m=+927.862365024" watchObservedRunningTime="2026-10-31 00:18:15.438540787 +0000 UTC m=+927.913899210" Oct 31 00:18:15.765071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:15.765034 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:15.770609 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:15.770563 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:16.220082 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:16.220010 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:16.221159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:16.221137 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:17.235834 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.235800 4969 generic.go:356] "Generic (PLEG): container finished" podID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerID="978465d2019fa539b8ae78c61a671d84a4f6c9e0271965643d4a690c333ae113" exitCode=0 Oct 31 00:18:17.236390 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.235830 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" event={"ID":"d7f907a2-82d5-44ce-8d56-5c3de82f6b41","Type":"ContainerDied","Data":"978465d2019fa539b8ae78c61a671d84a4f6c9e0271965643d4a690c333ae113"} Oct 31 00:18:17.236464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.236432 4969 scope.go:122] "RemoveContainer" containerID="52c05a964c090d5bc554704dc5ecfa77f82b2c24fc4d157355fdec38dd524874" Oct 31 00:18:17.358180 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.358154 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:18:17.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.435782 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-envoy\") pod \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " Oct 31 00:18:17.436009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.435849 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-podinfo\") pod \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " Oct 31 00:18:17.436009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.435902 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-socket\") pod \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " Oct 31 00:18:17.436009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.435973 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-kube-api-access-fdq6q\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-kube-api-access-fdq6q\") pod \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " Oct 31 00:18:17.436179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436019 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-data\") pod \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " Oct 31 00:18:17.436179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436047 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istiod-ca-cert\") pod \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " Oct 31 00:18:17.436179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436098 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-token\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-token\") pod \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " Oct 31 00:18:17.436179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436131 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-certs\") pod \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " Oct 31 00:18:17.436179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436160 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-credential-socket\") pod \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\" (UID: \"d7f907a2-82d5-44ce-8d56-5c3de82f6b41\") " Oct 31 00:18:17.436424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436247 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-socket" pod "d7f907a2-82d5-44ce-8d56-5c3de82f6b41" (UID: "d7f907a2-82d5-44ce-8d56-5c3de82f6b41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:17.436542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436518 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istiod-ca-cert" pod "d7f907a2-82d5-44ce-8d56-5c3de82f6b41" (UID: "d7f907a2-82d5-44ce-8d56-5c3de82f6b41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:17.436632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436576 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:17.436632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436571 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-certs" pod "d7f907a2-82d5-44ce-8d56-5c3de82f6b41" (UID: "d7f907a2-82d5-44ce-8d56-5c3de82f6b41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:17.436720 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436650 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-data" pod "d7f907a2-82d5-44ce-8d56-5c3de82f6b41" (UID: "d7f907a2-82d5-44ce-8d56-5c3de82f6b41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:17.437011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.436989 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-credential-socket" pod "d7f907a2-82d5-44ce-8d56-5c3de82f6b41" (UID: "d7f907a2-82d5-44ce-8d56-5c3de82f6b41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:17.438349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.438330 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-podinfo" pod "d7f907a2-82d5-44ce-8d56-5c3de82f6b41" (UID: "d7f907a2-82d5-44ce-8d56-5c3de82f6b41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:18:17.438560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.438543 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-kube-api-access-fdq6q" pod "d7f907a2-82d5-44ce-8d56-5c3de82f6b41" (UID: "d7f907a2-82d5-44ce-8d56-5c3de82f6b41"). InnerVolumeSpecName "kube-api-access-fdq6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:17.438737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.438713 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-envoy" pod "d7f907a2-82d5-44ce-8d56-5c3de82f6b41" (UID: "d7f907a2-82d5-44ce-8d56-5c3de82f6b41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:17.438835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.438757 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-token" pod "d7f907a2-82d5-44ce-8d56-5c3de82f6b41" (UID: "d7f907a2-82d5-44ce-8d56-5c3de82f6b41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:17.538096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.538069 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:17.538096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.538093 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:17.538096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.538102 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:17.538321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.538111 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:17.538321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.538119 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:17.538321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.538129 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:17.538321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.538137 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:17.538321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:17.538145 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdq6q\" (UniqueName: \"kubernetes.io/projected/d7f907a2-82d5-44ce-8d56-5c3de82f6b41-kube-api-access-fdq6q\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:18.252512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:18.252475 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" event={"ID":"d7f907a2-82d5-44ce-8d56-5c3de82f6b41","Type":"ContainerDied","Data":"17cd34ccfa14ce7a6b60674b78ab4f255ee4d25fa0144a4ee8c09afb45ef193a"} Oct 31 00:18:18.252512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:18.252496 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9" Oct 31 00:18:18.253101 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:18.252572 4969 scope.go:122] "RemoveContainer" containerID="978465d2019fa539b8ae78c61a671d84a4f6c9e0271965643d4a690c333ae113" Oct 31 00:18:18.469037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:18.469002 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9"] Oct 31 00:18:18.473928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:18.473903 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-15eab294-openshift-default-649cdc68c5-gd8p9"] Oct 31 00:18:18.634335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:18.634301 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx"] Oct 31 00:18:18.634636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:18.634576 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" containerID="cri-o://c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1" gracePeriod=30 Oct 31 00:18:19.733559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:19.733524 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" path="/var/lib/kubelet/pods/d7f907a2-82d5-44ce-8d56-5c3de82f6b41/volumes" Oct 31 00:18:20.145577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.145542 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c"] Oct 31 00:18:20.146278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.146251 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" Oct 31 00:18:20.146278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.146280 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" Oct 31 00:18:20.146430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.146294 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" Oct 31 00:18:20.146430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.146301 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" Oct 31 00:18:20.146490 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.146446 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" Oct 31 00:18:20.146490 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.146456 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f907a2-82d5-44ce-8d56-5c3de82f6b41" containerName="istio-proxy" Oct 31 00:18:20.151806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.151780 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.206579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.206546 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-869cca-openshift-default-dockercfg-49wg8\"" Oct 31 00:18:20.259094 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.259064 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c"] Oct 31 00:18:20.265645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.265616 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.269195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.269147 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.269441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.269416 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.269561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.269467 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmh7s\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-kube-api-access-xmh7s\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.269726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.269702 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.269894 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.269856 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.270024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.269928 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.270075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.270033 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b5e359-4787-45df-b48e-b83fabb5afb9-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.270127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.270088 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.370936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.370899 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b5e359-4787-45df-b48e-b83fabb5afb9-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.370949 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.370972 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.370987 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371003 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371025 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmh7s\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-kube-api-access-xmh7s\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371090 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371130 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371169 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371526 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371491 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371526 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371516 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371496 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371573 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.371793 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.371767 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b5e359-4787-45df-b48e-b83fabb5afb9-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.373635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.373584 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.373736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.373638 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.431707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.431639 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.431866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.431844 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmh7s\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-kube-api-access-xmh7s\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.465735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.465709 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:20.930678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.930654 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c"] Oct 31 00:18:20.932098 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:18:20.932072 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b5e359_4787_45df_b48e_b83fabb5afb9.slice/crio-0f82941aea6d8ddb47ff4f66c5afb68ba487e43b84ed99ad951917b57874c56c WatchSource:0}: Error finding container 0f82941aea6d8ddb47ff4f66c5afb68ba487e43b84ed99ad951917b57874c56c: Status 404 returned error can't find the container with id 0f82941aea6d8ddb47ff4f66c5afb68ba487e43b84ed99ad951917b57874c56c Oct 31 00:18:20.934387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.934350 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:20.934503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.934427 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:20.934503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:20.934473 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:21.293668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:21.292725 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" event={"ID":"18b5e359-4787-45df-b48e-b83fabb5afb9","Type":"ContainerStarted","Data":"ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c"} Oct 31 00:18:21.293668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:21.292800 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" event={"ID":"18b5e359-4787-45df-b48e-b83fabb5afb9","Type":"ContainerStarted","Data":"0f82941aea6d8ddb47ff4f66c5afb68ba487e43b84ed99ad951917b57874c56c"} Oct 31 00:18:21.466152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:21.466121 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:22.490926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:22.490853 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:22.611548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:22.611489 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" podStartSLOduration=2.611469365 podStartE2EDuration="2.611469365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:21.416785813 +0000 UTC m=+933.892144236" watchObservedRunningTime="2026-10-31 00:18:22.611469365 +0000 UTC m=+935.086827785" Oct 31 00:18:23.326449 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:23.326416 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:23.327445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:23.327420 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:18:23.922201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:23.922168 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:18:24.032739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.032709 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-istio-token\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-istio-token\") pod \"468c0ca1-e5de-4634-867b-017cb193eb36\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " Oct 31 00:18:24.032903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.032766 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468c0ca1-e5de-4634-867b-017cb193eb36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468c0ca1-e5de-4634-867b-017cb193eb36-istiod-ca-cert\") pod \"468c0ca1-e5de-4634-867b-017cb193eb36\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " Oct 31 00:18:24.032903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.032800 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-data\") pod \"468c0ca1-e5de-4634-867b-017cb193eb36\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " Oct 31 00:18:24.032903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.032821 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/468c0ca1-e5de-4634-867b-017cb193eb36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468c0ca1-e5de-4634-867b-017cb193eb36-istio-podinfo\") pod \"468c0ca1-e5de-4634-867b-017cb193eb36\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " Oct 31 00:18:24.032903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.032868 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-kube-api-access-qm5rp\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-kube-api-access-qm5rp\") pod \"468c0ca1-e5de-4634-867b-017cb193eb36\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " Oct 31 00:18:24.033143 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.032961 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-socket\") pod \"468c0ca1-e5de-4634-867b-017cb193eb36\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " Oct 31 00:18:24.033143 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.032996 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-certs\") pod \"468c0ca1-e5de-4634-867b-017cb193eb36\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " Oct 31 00:18:24.033143 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033033 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-envoy\") pod \"468c0ca1-e5de-4634-867b-017cb193eb36\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " Oct 31 00:18:24.033143 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033061 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-credential-socket\") pod \"468c0ca1-e5de-4634-867b-017cb193eb36\" (UID: \"468c0ca1-e5de-4634-867b-017cb193eb36\") " Oct 31 00:18:24.033343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033168 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-data" pod "468c0ca1-e5de-4634-867b-017cb193eb36" (UID: "468c0ca1-e5de-4634-867b-017cb193eb36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:24.033404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033329 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-certs" pod "468c0ca1-e5de-4634-867b-017cb193eb36" (UID: "468c0ca1-e5de-4634-867b-017cb193eb36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:24.033404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033347 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468c0ca1-e5de-4634-867b-017cb193eb36-istiod-ca-cert" pod "468c0ca1-e5de-4634-867b-017cb193eb36" (UID: "468c0ca1-e5de-4634-867b-017cb193eb36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:24.033404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033347 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-socket" pod "468c0ca1-e5de-4634-867b-017cb193eb36" (UID: "468c0ca1-e5de-4634-867b-017cb193eb36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:24.033605 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033557 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468c0ca1-e5de-4634-867b-017cb193eb36-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.033728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033582 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.033728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033621 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.033728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033636 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.033728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.033642 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-credential-socket" pod "468c0ca1-e5de-4634-867b-017cb193eb36" (UID: "468c0ca1-e5de-4634-867b-017cb193eb36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:24.035859 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.035835 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/468c0ca1-e5de-4634-867b-017cb193eb36-istio-podinfo" pod "468c0ca1-e5de-4634-867b-017cb193eb36" (UID: "468c0ca1-e5de-4634-867b-017cb193eb36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:18:24.035968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.035883 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-istio-token" pod "468c0ca1-e5de-4634-867b-017cb193eb36" (UID: "468c0ca1-e5de-4634-867b-017cb193eb36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:24.036038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.036016 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-kube-api-access-qm5rp" pod "468c0ca1-e5de-4634-867b-017cb193eb36" (UID: "468c0ca1-e5de-4634-867b-017cb193eb36"). InnerVolumeSpecName "kube-api-access-qm5rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:24.036115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.036097 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-envoy" pod "468c0ca1-e5de-4634-867b-017cb193eb36" (UID: "468c0ca1-e5de-4634-867b-017cb193eb36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:18:24.134394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.134359 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.134394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.134388 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468c0ca1-e5de-4634-867b-017cb193eb36-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.134394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.134398 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm5rp\" (UniqueName: \"kubernetes.io/projected/468c0ca1-e5de-4634-867b-017cb193eb36-kube-api-access-qm5rp\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.134660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.134409 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.134660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.134418 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468c0ca1-e5de-4634-867b-017cb193eb36-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:18:24.334038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.333961 4969 generic.go:356] "Generic (PLEG): container finished" podID="468c0ca1-e5de-4634-867b-017cb193eb36" containerID="c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1" exitCode=0 Oct 31 00:18:24.334038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.334027 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" Oct 31 00:18:24.334213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.334035 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" event={"ID":"468c0ca1-e5de-4634-867b-017cb193eb36","Type":"ContainerDied","Data":"c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1"} Oct 31 00:18:24.334213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.334087 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx" event={"ID":"468c0ca1-e5de-4634-867b-017cb193eb36","Type":"ContainerDied","Data":"5d2274a7599fad23621db8c65c82239d2d9bbd1f7f77dc8eedd64c27b0d425ab"} Oct 31 00:18:24.334213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.334108 4969 scope.go:122] "RemoveContainer" containerID="c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1" Oct 31 00:18:24.344745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.344720 4969 scope.go:122] "RemoveContainer" containerID="c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5" Oct 31 00:18:24.354048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.354028 4969 scope.go:122] "RemoveContainer" containerID="c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1" Oct 31 00:18:24.354374 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:18:24.354354 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1\": container with ID starting with c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1 not found: ID does not exist" containerID="c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1" Oct 31 00:18:24.354439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.354386 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1"} err="failed to get container status \"c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1\": rpc error: code = NotFound desc = could not find container \"c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1\": container with ID starting with c07302c9c598e324670f7b76b6c5e841d15e5d21968b8e6d6a9d476dafae1ac1 not found: ID does not exist" Oct 31 00:18:24.354439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.354411 4969 scope.go:122] "RemoveContainer" containerID="c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5" Oct 31 00:18:24.354690 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:18:24.354668 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5\": container with ID starting with c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5 not found: ID does not exist" containerID="c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5" Oct 31 00:18:24.354835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.354698 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5"} err="failed to get container status \"c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5\": rpc error: code = NotFound desc = could not find container \"c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5\": container with ID starting with c7ebfe8a55eb4980702fce36e70483e004eda40a4d68e8a364905441a4683dd5 not found: ID does not exist" Oct 31 00:18:24.513275 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.513240 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx"] Oct 31 00:18:24.562722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:24.562687 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b83a178e-openshift-default-8475f7654f-v6djx"] Oct 31 00:18:25.342247 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:25.342151 4969 generic.go:356] "Generic (PLEG): container finished" podID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerID="8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708" exitCode=0 Oct 31 00:18:25.342758 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:25.342232 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" event={"ID":"6c0668f1-3bb4-4cad-976e-9b97244f5622","Type":"ContainerDied","Data":"8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708"} Oct 31 00:18:25.342758 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:25.342681 4969 scope.go:122] "RemoveContainer" containerID="8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708" Oct 31 00:18:25.343774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:25.343751 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:25.343832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:25.343822 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:25.343866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:25.343851 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:25.734327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:25.734282 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" path="/var/lib/kubelet/pods/468c0ca1-e5de-4634-867b-017cb193eb36/volumes" Oct 31 00:18:26.358731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:26.358696 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" event={"ID":"6c0668f1-3bb4-4cad-976e-9b97244f5622","Type":"ContainerStarted","Data":"993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b"} Oct 31 00:18:26.469511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:26.469476 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:18:26.470376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:26.470354 4969 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:18:26.470470 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:26.470417 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:18:27.470236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:27.470208 4969 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:18:27.470768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:27.470272 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:18:27.951375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:27.951338 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85"] Oct 31 00:18:27.951853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:27.951838 4969 cpu_manager.go:414] "containerMap: removing container" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" Oct 31 00:18:27.951911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:27.951854 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" Oct 31 00:18:27.951967 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:27.951958 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" Oct 31 00:18:27.952010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:27.951968 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" Oct 31 00:18:27.956315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:27.956301 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:28.016180 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.016147 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85"] Oct 31 00:18:28.058277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.058248 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-869cca\"" Oct 31 00:18:28.116539 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.116500 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f292z\" (UniqueName: \"kubernetes.io/projected/814a5540-13fc-4f69-b185-f71fea2f7fdd-kube-api-access-f292z\") pod \"payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85\" (UID: \"814a5540-13fc-4f69-b185-f71fea2f7fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:28.116865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.116840 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814a5540-13fc-4f69-b185-f71fea2f7fdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85\" (UID: \"814a5540-13fc-4f69-b185-f71fea2f7fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:28.125014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.124988 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc"] Oct 31 00:18:28.125694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.125666 4969 cpu_manager.go:414] "containerMap: removing container" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" Oct 31 00:18:28.125694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.125686 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468c0ca1-e5de-4634-867b-017cb193eb36" containerName="istio-proxy" Oct 31 00:18:28.129111 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.129096 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:28.197619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.177541 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc"] Oct 31 00:18:28.197619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.182912 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-869cca-dockercfg-447t2\"" Oct 31 00:18:28.226333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.226246 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f292z\" (UniqueName: \"kubernetes.io/projected/814a5540-13fc-4f69-b185-f71fea2f7fdd-kube-api-access-f292z\") pod \"payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85\" (UID: \"814a5540-13fc-4f69-b185-f71fea2f7fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:28.226616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.226573 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814a5540-13fc-4f69-b185-f71fea2f7fdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85\" (UID: \"814a5540-13fc-4f69-b185-f71fea2f7fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:28.227708 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.227684 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814a5540-13fc-4f69-b185-f71fea2f7fdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85\" (UID: \"814a5540-13fc-4f69-b185-f71fea2f7fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:28.302573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.302498 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f292z\" (UniqueName: \"kubernetes.io/projected/814a5540-13fc-4f69-b185-f71fea2f7fdd-kube-api-access-f292z\") pod \"payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85\" (UID: \"814a5540-13fc-4f69-b185-f71fea2f7fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:28.328077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.328050 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc\" (UID: \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:28.328223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.328084 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxwqb\" (UniqueName: \"kubernetes.io/projected/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-kube-api-access-qxwqb\") pod \"payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc\" (UID: \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:28.429114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.429084 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc\" (UID: \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:28.429114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.429121 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxwqb\" (UniqueName: \"kubernetes.io/projected/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-kube-api-access-qxwqb\") pod \"payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc\" (UID: \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:28.429813 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.429793 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc\" (UID: \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:28.469874 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.469848 4969 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:18:28.469983 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.469894 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:18:28.502422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.502324 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxwqb\" (UniqueName: \"kubernetes.io/projected/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-kube-api-access-qxwqb\") pod \"payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc\" (UID: \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:28.569878 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.569850 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:28.742670 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.742634 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:28.972144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:28.972120 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85"] Oct 31 00:18:28.973016 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:18:28.972992 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814a5540_13fc_4f69_b185_f71fea2f7fdd.slice/crio-32538084202f70fdf653a9ae011e4d13219b04acc1c8dc4d3cecd6d78e50dd75 WatchSource:0}: Error finding container 32538084202f70fdf653a9ae011e4d13219b04acc1c8dc4d3cecd6d78e50dd75: Status 404 returned error can't find the container with id 32538084202f70fdf653a9ae011e4d13219b04acc1c8dc4d3cecd6d78e50dd75 Oct 31 00:18:29.004651 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.001446 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:18:29.141540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.141512 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc"] Oct 31 00:18:29.142645 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:18:29.142616 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bf1d7a_2f8b_4463_9e58_3cf37e1beed3.slice/crio-afca777a8782a08d824c0d61ac1cec58a763b78bd109d6e53b4c7f8868a62805 WatchSource:0}: Error finding container afca777a8782a08d824c0d61ac1cec58a763b78bd109d6e53b4c7f8868a62805: Status 404 returned error can't find the container with id afca777a8782a08d824c0d61ac1cec58a763b78bd109d6e53b4c7f8868a62805 Oct 31 00:18:29.414355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.414308 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" event={"ID":"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3","Type":"ContainerStarted","Data":"ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0"} Oct 31 00:18:29.414548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.414422 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" event={"ID":"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3","Type":"ContainerStarted","Data":"afca777a8782a08d824c0d61ac1cec58a763b78bd109d6e53b4c7f8868a62805"} Oct 31 00:18:29.414803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.414779 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:29.416997 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.416969 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" event={"ID":"814a5540-13fc-4f69-b185-f71fea2f7fdd","Type":"ContainerStarted","Data":"13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd"} Oct 31 00:18:29.417110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.417013 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" event={"ID":"814a5540-13fc-4f69-b185-f71fea2f7fdd","Type":"ContainerStarted","Data":"32538084202f70fdf653a9ae011e4d13219b04acc1c8dc4d3cecd6d78e50dd75"} Oct 31 00:18:29.417110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.417071 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:29.469482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.469452 4969 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:18:29.469635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.469501 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:18:29.532358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.532308 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" podStartSLOduration=1.532292539 podStartE2EDuration="1.532292539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:29.530739475 +0000 UTC m=+942.006097898" watchObservedRunningTime="2026-10-31 00:18:29.532292539 +0000 UTC m=+942.007650961" Oct 31 00:18:29.659581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:29.659528 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" podStartSLOduration=2.659513589 podStartE2EDuration="2.659513589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:29.656995955 +0000 UTC m=+942.132354378" watchObservedRunningTime="2026-10-31 00:18:29.659513589 +0000 UTC m=+942.134872011" Oct 31 00:18:30.470315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:30.470283 4969 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:18:30.470484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:30.470331 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:18:31.512145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:31.512101 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:18:31.513740 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:31.513710 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 00:18:32.726993 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:18:32.726959 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:18:40.427064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:40.427036 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 00:18:40.427460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:40.427102 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 00:18:57.608510 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:57.608476 4969 generic.go:356] "Generic (PLEG): container finished" podID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerID="768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b" exitCode=0 Oct 31 00:18:57.609105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:57.608557 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" event={"ID":"f32982e9-15ef-45ac-ab4c-55f742b5bc4c","Type":"ContainerDied","Data":"768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b"} Oct 31 00:18:57.609105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:57.609046 4969 scope.go:122] "RemoveContainer" containerID="768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b" Oct 31 00:18:57.610411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:57.610387 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:57.610479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:57.610452 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:57.610518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:57.610483 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:18:58.631119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:58.631081 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" event={"ID":"f32982e9-15ef-45ac-ab4c-55f742b5bc4c","Type":"ContainerStarted","Data":"6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa"} Oct 31 00:18:58.765005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:58.764971 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:18:58.766296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:58.766268 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:18:58.766436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:58.766329 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:18:59.764732 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:59.764698 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:18:59.765121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:18:59.764760 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:19:00.764807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:00.764775 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:19:00.765189 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:00.764819 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:19:01.226928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:01.226708 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:19:01.765333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:01.765297 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:19:01.765733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:01.765360 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:19:02.751699 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:19:02.751662 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:19:02.765728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:02.765694 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:19:02.766197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:02.765752 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:19:03.865613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:03.865553 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:19:03.866880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:03.866859 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:19:06.717434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.717385 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75"] Oct 31 00:19:06.722230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.722205 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.786641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.786582 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-model-cache\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.786896 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.786873 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-home\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.787034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.787019 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-tmp-dir\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.787195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.787177 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5524343d-7ab6-4683-bf9b-c2658d396a49-tls-certs\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.787323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.787307 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-dshm\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.787472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.787453 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2g8b\" (UniqueName: \"kubernetes.io/projected/5524343d-7ab6-4683-bf9b-c2658d396a49-kube-api-access-d2g8b\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.795521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.795490 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75"] Oct 31 00:19:06.824885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.824861 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"default-dockercfg-2clwg\"" Oct 31 00:19:06.825079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.824961 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"openshift-service-ca.crt\"" Oct 31 00:19:06.825161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.824946 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"kube-root-ca.crt\"" Oct 31 00:19:06.825161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.824996 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"simulator-w0-10a7aa-kserve-self-signed-certs\"" Oct 31 00:19:06.888279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.888244 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2g8b\" (UniqueName: \"kubernetes.io/projected/5524343d-7ab6-4683-bf9b-c2658d396a49-kube-api-access-d2g8b\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.888453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.888341 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-model-cache\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.888453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.888361 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-home\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.888453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.888381 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-tmp-dir\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.888453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.888437 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5524343d-7ab6-4683-bf9b-c2658d396a49-tls-certs\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.888669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.888470 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-dshm\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.888832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.888810 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-home\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.888984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.888853 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-model-cache\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.888984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.888966 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-tmp-dir\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.890926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.890900 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-dshm\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.891190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.891176 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5524343d-7ab6-4683-bf9b-c2658d396a49-tls-certs\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:06.958428 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:06.958395 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2g8b\" (UniqueName: \"kubernetes.io/projected/5524343d-7ab6-4683-bf9b-c2658d396a49-kube-api-access-d2g8b\") pod \"simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:07.040167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.040089 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:07.421243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.421212 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75"] Oct 31 00:19:07.422401 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:19:07.422371 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5524343d_7ab6_4683_bf9b_c2658d396a49.slice/crio-e28dd46bac64b19fc6ece433c15548d0f91f4ee6fe2367420f9ec448b81ce749 WatchSource:0}: Error finding container e28dd46bac64b19fc6ece433c15548d0f91f4ee6fe2367420f9ec448b81ce749: Status 404 returned error can't find the container with id e28dd46bac64b19fc6ece433c15548d0f91f4ee6fe2367420f9ec448b81ce749 Oct 31 00:19:07.695279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.695201 4969 generic.go:356] "Generic (PLEG): container finished" podID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerID="ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c" exitCode=0 Oct 31 00:19:07.695456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.695272 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" event={"ID":"18b5e359-4787-45df-b48e-b83fabb5afb9","Type":"ContainerDied","Data":"ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c"} Oct 31 00:19:07.695676 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.695658 4969 scope.go:122] "RemoveContainer" containerID="ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c" Oct 31 00:19:07.696988 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.696955 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:19:07.697088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.697045 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:19:07.697088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.697083 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:19:07.697182 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.697160 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" event={"ID":"5524343d-7ab6-4683-bf9b-c2658d396a49","Type":"ContainerStarted","Data":"25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6"} Oct 31 00:19:07.697217 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.697189 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" event={"ID":"5524343d-7ab6-4683-bf9b-c2658d396a49","Type":"ContainerStarted","Data":"e28dd46bac64b19fc6ece433c15548d0f91f4ee6fe2367420f9ec448b81ce749"} Oct 31 00:19:07.935554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:07.935489 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" podStartSLOduration=1.93547043 podStartE2EDuration="1.93547043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:07.934654151 +0000 UTC m=+980.410012575" watchObservedRunningTime="2026-10-31 00:19:07.93547043 +0000 UTC m=+980.410828854" Oct 31 00:19:08.040738 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:08.040672 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:08.327080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:08.326994 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:19:08.709647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:08.709585 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" event={"ID":"18b5e359-4787-45df-b48e-b83fabb5afb9","Type":"ContainerStarted","Data":"c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471"} Oct 31 00:19:09.465988 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:09.465952 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:19:09.467031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:09.467009 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:19:09.467157 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:09.467066 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:19:10.466648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:10.466579 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:19:10.467118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:10.466688 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:19:11.466392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:11.466355 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:19:11.466559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:11.466420 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:19:12.466062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:12.466024 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:19:12.466460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:12.466079 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:19:13.466326 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:13.466290 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:19:13.466764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:13.466359 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:19:14.470545 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:14.470518 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:19:14.471033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:14.471011 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:19:14.471687 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:14.471670 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:19:17.040576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:17.040545 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:17.053189 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:17.053165 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:17.778374 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:17.778348 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:19:32.727042 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:19:32.727004 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:19:42.750815 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.750781 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk"] Oct 31 00:19:42.755049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.755025 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.849117 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.849079 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-model-cache\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.849296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.849179 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-dshm\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.849296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.849216 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tmp-dir\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.849385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.849315 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22phv\" (UniqueName: \"kubernetes.io/projected/ebd96c4a-8321-47f2-88d1-1e2b281ab675-kube-api-access-22phv\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.849385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.849355 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-home\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.849642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.849584 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tls-certs\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.859666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.859644 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"openshift-service-ca.crt\"" Oct 31 00:19:42.859666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.859657 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"default-dockercfg-nb88j\"" Oct 31 00:19:42.859826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.859668 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"simulator-w2-e1ca71-kserve-self-signed-certs\"" Oct 31 00:19:42.859826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.859661 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"kube-root-ca.crt\"" Oct 31 00:19:42.862733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.862713 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk"] Oct 31 00:19:42.950398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.950363 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tls-certs\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.950628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.950403 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-model-cache\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.950628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.950449 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-dshm\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.950628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.950476 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tmp-dir\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.950628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.950524 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22phv\" (UniqueName: \"kubernetes.io/projected/ebd96c4a-8321-47f2-88d1-1e2b281ab675-kube-api-access-22phv\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.950628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.950551 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-home\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.950985 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.950956 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-model-cache\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.951073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.951020 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tmp-dir\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.951073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.951047 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-home\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.952952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.952929 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-dshm\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:42.953116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:42.953097 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tls-certs\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:43.011846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:43.011779 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22phv\" (UniqueName: \"kubernetes.io/projected/ebd96c4a-8321-47f2-88d1-1e2b281ab675-kube-api-access-22phv\") pod \"simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:43.067292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:43.067267 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:43.492019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:43.491977 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk"] Oct 31 00:19:43.936064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:43.936023 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" event={"ID":"ebd96c4a-8321-47f2-88d1-1e2b281ab675","Type":"ContainerStarted","Data":"68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc"} Oct 31 00:19:43.936064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:43.936069 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" event={"ID":"ebd96c4a-8321-47f2-88d1-1e2b281ab675","Type":"ContainerStarted","Data":"b0916847b0be4b9fcee9891c5c8050980f950634790a9113509d675854e6b579"} Oct 31 00:19:44.053907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:44.053853 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podStartSLOduration=2.053837566 podStartE2EDuration="2.053837566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:44.052231498 +0000 UTC m=+1016.527589922" watchObservedRunningTime="2026-10-31 00:19:44.053837566 +0000 UTC m=+1016.529195989" Oct 31 00:19:44.067669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:44.067648 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:53.068219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:53.068177 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:53.080696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:53.080670 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:19:54.011992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:19:54.011961 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:20:02.727250 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:20:02.727166 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:20:27.775338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:27.775306 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b"] Oct 31 00:20:27.779295 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:27.779277 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:27.883248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:27.883215 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:20:27.883419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:27.883324 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:20:27.886376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:27.886358 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b"] Oct 31 00:20:27.910446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:27.910412 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fa72829-cr77b\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:27.910619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:27.910459 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fa72829-cr77b\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:27.910679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:27.910652 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jknc5\" (UniqueName: \"kubernetes.io/projected/00829b05-9424-4102-bfcc-2a726893b7f3-kube-api-access-jknc5\") pod \"maas-api-revoke-sub-3fa72829-cr77b\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:28.012305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:28.012276 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fa72829-cr77b\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:28.012469 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:28.012316 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fa72829-cr77b\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:28.012469 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:28.012443 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jknc5\" (UniqueName: \"kubernetes.io/projected/00829b05-9424-4102-bfcc-2a726893b7f3-kube-api-access-jknc5\") pod \"maas-api-revoke-sub-3fa72829-cr77b\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:28.013107 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:28.013044 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fa72829-cr77b\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:28.014806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:28.014784 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fa72829-cr77b\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:28.076503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:28.076435 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jknc5\" (UniqueName: \"kubernetes.io/projected/00829b05-9424-4102-bfcc-2a726893b7f3-kube-api-access-jknc5\") pod \"maas-api-revoke-sub-3fa72829-cr77b\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:28.092231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:28.092207 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:28.471059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:28.471032 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b"] Oct 31 00:20:28.472744 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:20:28.472718 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00829b05_9424_4102_bfcc_2a726893b7f3.slice/crio-f1e565805146842db1f5d8226beb71283a856f607cd3230b6619ebdcdf5b15fe WatchSource:0}: Error finding container f1e565805146842db1f5d8226beb71283a856f607cd3230b6619ebdcdf5b15fe: Status 404 returned error can't find the container with id f1e565805146842db1f5d8226beb71283a856f607cd3230b6619ebdcdf5b15fe Oct 31 00:20:29.231284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:29.231251 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" event={"ID":"00829b05-9424-4102-bfcc-2a726893b7f3","Type":"ContainerStarted","Data":"e3f127b60a1fca17f3c9b07ab7b195c7a9fc978b699fd43716f9f519a19aadf2"} Oct 31 00:20:29.231731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:29.231304 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" event={"ID":"00829b05-9424-4102-bfcc-2a726893b7f3","Type":"ContainerStarted","Data":"f1e565805146842db1f5d8226beb71283a856f607cd3230b6619ebdcdf5b15fe"} Oct 31 00:20:29.352271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:29.352218 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" podStartSLOduration=2.352204216 podStartE2EDuration="2.352204216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:29.347108702 +0000 UTC m=+1061.822467126" watchObservedRunningTime="2026-10-31 00:20:29.352204216 +0000 UTC m=+1061.827562632" Oct 31 00:20:30.238761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:30.238729 4969 generic.go:356] "Generic (PLEG): container finished" podID="00829b05-9424-4102-bfcc-2a726893b7f3" containerID="e3f127b60a1fca17f3c9b07ab7b195c7a9fc978b699fd43716f9f519a19aadf2" exitCode=0 Oct 31 00:20:30.239135 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:30.238814 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" event={"ID":"00829b05-9424-4102-bfcc-2a726893b7f3","Type":"ContainerDied","Data":"e3f127b60a1fca17f3c9b07ab7b195c7a9fc978b699fd43716f9f519a19aadf2"} Oct 31 00:20:31.386615 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.386562 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:31.548557 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.548480 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00829b05-9424-4102-bfcc-2a726893b7f3-kube-api-access-jknc5\" (UniqueName: \"kubernetes.io/projected/00829b05-9424-4102-bfcc-2a726893b7f3-kube-api-access-jknc5\") pod \"00829b05-9424-4102-bfcc-2a726893b7f3\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " Oct 31 00:20:31.548747 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.548581 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-xks-ca\") pod \"00829b05-9424-4102-bfcc-2a726893b7f3\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " Oct 31 00:20:31.548747 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.548690 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-openshift-ca\") pod \"00829b05-9424-4102-bfcc-2a726893b7f3\" (UID: \"00829b05-9424-4102-bfcc-2a726893b7f3\") " Oct 31 00:20:31.549028 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.549002 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-openshift-ca" pod "00829b05-9424-4102-bfcc-2a726893b7f3" (UID: "00829b05-9424-4102-bfcc-2a726893b7f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:31.549110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.549082 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:20:31.550847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.550823 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00829b05-9424-4102-bfcc-2a726893b7f3-kube-api-access-jknc5" pod "00829b05-9424-4102-bfcc-2a726893b7f3" (UID: "00829b05-9424-4102-bfcc-2a726893b7f3"). InnerVolumeSpecName "kube-api-access-jknc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:31.550847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.550844 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-xks-ca" pod "00829b05-9424-4102-bfcc-2a726893b7f3" (UID: "00829b05-9424-4102-bfcc-2a726893b7f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:31.650436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.650410 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jknc5\" (UniqueName: \"kubernetes.io/projected/00829b05-9424-4102-bfcc-2a726893b7f3-kube-api-access-jknc5\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:20:31.650436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:31.650436 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00829b05-9424-4102-bfcc-2a726893b7f3-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:20:32.253787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:32.253753 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" Oct 31 00:20:32.253787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:32.253786 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b" event={"ID":"00829b05-9424-4102-bfcc-2a726893b7f3","Type":"ContainerDied","Data":"f1e565805146842db1f5d8226beb71283a856f607cd3230b6619ebdcdf5b15fe"} Oct 31 00:20:32.254014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:32.253839 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e565805146842db1f5d8226beb71283a856f607cd3230b6619ebdcdf5b15fe" Oct 31 00:20:32.726998 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:20:32.726955 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:20:42.138167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.138132 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26"] Oct 31 00:20:42.138900 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.138880 4969 cpu_manager.go:414] "containerMap: removing container" podUID="00829b05-9424-4102-bfcc-2a726893b7f3" containerName="revoke-keys" Oct 31 00:20:42.138993 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.138901 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00829b05-9424-4102-bfcc-2a726893b7f3" containerName="revoke-keys" Oct 31 00:20:42.139095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.139081 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="00829b05-9424-4102-bfcc-2a726893b7f3" containerName="revoke-keys" Oct 31 00:20:42.142288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.142266 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.216856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.216829 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26"] Oct 31 00:20:42.244015 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.243985 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-home\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.244144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.244111 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tls-certs\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.244185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.244143 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-model-cache\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.244234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.244212 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6jh\" (UniqueName: \"kubernetes.io/projected/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-kube-api-access-ck6jh\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.244273 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.244250 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tmp-dir\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.244306 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.244281 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-dshm\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.246500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.246484 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"premium-w2-e1ca71-kserve-self-signed-certs\"" Oct 31 00:20:42.345065 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.345031 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tls-certs\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.345065 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.345066 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-model-cache\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.345295 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.345089 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6jh\" (UniqueName: \"kubernetes.io/projected/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-kube-api-access-ck6jh\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.345295 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.345121 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tmp-dir\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.345295 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.345149 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-dshm\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.345441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.345337 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-home\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.345521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.345502 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-model-cache\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.345619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.345572 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tmp-dir\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.345686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.345650 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-home\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.347931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.347901 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-dshm\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.348045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.348011 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tls-certs\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.414526 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.414456 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6jh\" (UniqueName: \"kubernetes.io/projected/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-kube-api-access-ck6jh\") pod \"premium-w2-e1ca71-kserve-549b9556d6-zmc26\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.453601 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.453575 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:42.852724 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:42.852696 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26"] Oct 31 00:20:42.854043 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:20:42.854014 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod525b9ee7_9d4e_48e6_b5e8_653b18b656d4.slice/crio-1e6212fab3bff969ed2e46f35f22d8f25915cfb467594f7c8d729c891bb1d27a WatchSource:0}: Error finding container 1e6212fab3bff969ed2e46f35f22d8f25915cfb467594f7c8d729c891bb1d27a: Status 404 returned error can't find the container with id 1e6212fab3bff969ed2e46f35f22d8f25915cfb467594f7c8d729c891bb1d27a Oct 31 00:20:43.326174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:43.326136 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" event={"ID":"525b9ee7-9d4e-48e6-b5e8-653b18b656d4","Type":"ContainerStarted","Data":"17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b"} Oct 31 00:20:43.326561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:43.326187 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" event={"ID":"525b9ee7-9d4e-48e6-b5e8-653b18b656d4","Type":"ContainerStarted","Data":"1e6212fab3bff969ed2e46f35f22d8f25915cfb467594f7c8d729c891bb1d27a"} Oct 31 00:20:43.454704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:43.454675 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:43.474177 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:43.474124 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podStartSLOduration=1.474106419 podStartE2EDuration="1.474106419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:43.470985586 +0000 UTC m=+1075.946344010" watchObservedRunningTime="2026-10-31 00:20:43.474106419 +0000 UTC m=+1075.949464842" Oct 31 00:20:45.904616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:45.904559 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs"] Oct 31 00:20:45.908447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:45.908432 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:45.977690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:45.977659 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs"] Oct 31 00:20:45.984767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:45.984741 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f688568f-5rzrs\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:45.984924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:45.984884 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7gvx\" (UniqueName: \"kubernetes.io/projected/d3c3b141-91f6-42c2-a388-5daf4ac59b87-kube-api-access-h7gvx\") pod \"maas-api-revoke-sub-f688568f-5rzrs\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:45.984980 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:45.984955 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f688568f-5rzrs\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:46.014430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.014400 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:20:46.014430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.014402 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:20:46.086066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.086031 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f688568f-5rzrs\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:46.086254 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.086135 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7gvx\" (UniqueName: \"kubernetes.io/projected/d3c3b141-91f6-42c2-a388-5daf4ac59b87-kube-api-access-h7gvx\") pod \"maas-api-revoke-sub-f688568f-5rzrs\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:46.086254 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.086207 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f688568f-5rzrs\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:46.086867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.086848 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f688568f-5rzrs\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:46.088468 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.088448 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f688568f-5rzrs\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:46.148730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.148705 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7gvx\" (UniqueName: \"kubernetes.io/projected/d3c3b141-91f6-42c2-a388-5daf4ac59b87-kube-api-access-h7gvx\") pod \"maas-api-revoke-sub-f688568f-5rzrs\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:46.224664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.224579 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:46.600437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:46.600405 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs"] Oct 31 00:20:46.601747 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:20:46.601717 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c3b141_91f6_42c2_a388_5daf4ac59b87.slice/crio-6d26cf0eccebbfee0345191d1f86fa5c1247fd3e42bf3ae2c278374c1b56cb55 WatchSource:0}: Error finding container 6d26cf0eccebbfee0345191d1f86fa5c1247fd3e42bf3ae2c278374c1b56cb55: Status 404 returned error can't find the container with id 6d26cf0eccebbfee0345191d1f86fa5c1247fd3e42bf3ae2c278374c1b56cb55 Oct 31 00:20:47.353538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:47.353438 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" event={"ID":"d3c3b141-91f6-42c2-a388-5daf4ac59b87","Type":"ContainerStarted","Data":"178537bebab7b504160cecfbf05f87083f33ce271f0457e6857a7f9c8d4a3668"} Oct 31 00:20:47.353918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:47.353544 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" event={"ID":"d3c3b141-91f6-42c2-a388-5daf4ac59b87","Type":"ContainerStarted","Data":"6d26cf0eccebbfee0345191d1f86fa5c1247fd3e42bf3ae2c278374c1b56cb55"} Oct 31 00:20:47.469921 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:47.469874 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" podStartSLOduration=2.469861092 podStartE2EDuration="2.469861092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:47.468415884 +0000 UTC m=+1079.943774307" watchObservedRunningTime="2026-10-31 00:20:47.469861092 +0000 UTC m=+1079.945219515" Oct 31 00:20:48.361312 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:48.361276 4969 generic.go:356] "Generic (PLEG): container finished" podID="d3c3b141-91f6-42c2-a388-5daf4ac59b87" containerID="178537bebab7b504160cecfbf05f87083f33ce271f0457e6857a7f9c8d4a3668" exitCode=0 Oct 31 00:20:48.361728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:48.361356 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" event={"ID":"d3c3b141-91f6-42c2-a388-5daf4ac59b87","Type":"ContainerDied","Data":"178537bebab7b504160cecfbf05f87083f33ce271f0457e6857a7f9c8d4a3668"} Oct 31 00:20:49.504151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.504127 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:49.629410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.629336 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-openshift-ca\") pod \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " Oct 31 00:20:49.629410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.629372 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-xks-ca\") pod \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " Oct 31 00:20:49.629579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.629461 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c3b141-91f6-42c2-a388-5daf4ac59b87-kube-api-access-h7gvx\" (UniqueName: \"kubernetes.io/projected/d3c3b141-91f6-42c2-a388-5daf4ac59b87-kube-api-access-h7gvx\") pod \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\" (UID: \"d3c3b141-91f6-42c2-a388-5daf4ac59b87\") " Oct 31 00:20:49.629722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.629701 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-openshift-ca" pod "d3c3b141-91f6-42c2-a388-5daf4ac59b87" (UID: "d3c3b141-91f6-42c2-a388-5daf4ac59b87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:49.629845 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.629820 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:20:49.631636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.631616 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c3b141-91f6-42c2-a388-5daf4ac59b87-kube-api-access-h7gvx" pod "d3c3b141-91f6-42c2-a388-5daf4ac59b87" (UID: "d3c3b141-91f6-42c2-a388-5daf4ac59b87"). InnerVolumeSpecName "kube-api-access-h7gvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:49.631727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.631706 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-xks-ca" pod "d3c3b141-91f6-42c2-a388-5daf4ac59b87" (UID: "d3c3b141-91f6-42c2-a388-5daf4ac59b87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:49.731293 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.731263 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c3b141-91f6-42c2-a388-5daf4ac59b87-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:20:49.731464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:49.731297 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7gvx\" (UniqueName: \"kubernetes.io/projected/d3c3b141-91f6-42c2-a388-5daf4ac59b87-kube-api-access-h7gvx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:20:50.384237 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:50.384206 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" Oct 31 00:20:50.384423 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:50.384245 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs" event={"ID":"d3c3b141-91f6-42c2-a388-5daf4ac59b87","Type":"ContainerDied","Data":"6d26cf0eccebbfee0345191d1f86fa5c1247fd3e42bf3ae2c278374c1b56cb55"} Oct 31 00:20:50.384423 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:50.384321 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d26cf0eccebbfee0345191d1f86fa5c1247fd3e42bf3ae2c278374c1b56cb55" Oct 31 00:20:52.454710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:52.454675 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:52.467688 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:52.467664 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:53.414684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:53.414662 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:20:59.960358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:59.960324 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg"] Oct 31 00:20:59.960839 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:59.960825 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d3c3b141-91f6-42c2-a388-5daf4ac59b87" containerName="revoke-keys" Oct 31 00:20:59.960887 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:59.960840 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c3b141-91f6-42c2-a388-5daf4ac59b87" containerName="revoke-keys" Oct 31 00:20:59.960959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:59.960949 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c3b141-91f6-42c2-a388-5daf4ac59b87" containerName="revoke-keys" Oct 31 00:20:59.964390 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:20:59.964366 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.024805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.024779 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-10a7aa-openshift-default-dockercfg-zqppq\"" Oct 31 00:21:00.077364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.077339 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg"] Oct 31 00:21:00.137981 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.137950 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.138153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.138015 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1047f6-6bc7-422a-9633-04fcfb13920c-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.138153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.138134 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.138258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.138197 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.138258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.138245 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.138370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.138299 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.138424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.138410 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.138482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.138471 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.138541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.138514 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nll\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-kube-api-access-s9nll\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239383 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239426 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nll\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-kube-api-access-s9nll\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239460 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239535 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1047f6-6bc7-422a-9633-04fcfb13920c-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239584 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239676 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239703 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239727 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239783 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.239872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.239867 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.240088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.240059 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.240144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.240109 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.240230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.240199 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.240497 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.240473 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1047f6-6bc7-422a-9633-04fcfb13920c-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.241960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.241936 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.242257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.242240 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.303264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.303236 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.303459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.303441 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nll\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-kube-api-access-s9nll\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:00.577717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:00.577637 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:01.034019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:01.033978 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg"] Oct 31 00:21:01.045927 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:21:01.045885 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1047f6_6bc7_422a_9633_04fcfb13920c.slice/crio-f59b8cf9d91792f5989e25773c93fe17efa29f788813f4e6e18182e10bb21852 WatchSource:0}: Error finding container f59b8cf9d91792f5989e25773c93fe17efa29f788813f4e6e18182e10bb21852: Status 404 returned error can't find the container with id f59b8cf9d91792f5989e25773c93fe17efa29f788813f4e6e18182e10bb21852 Oct 31 00:21:01.048308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:01.048278 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:21:01.048375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:01.048347 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:21:01.048422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:01.048375 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:21:01.460769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:01.460731 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" event={"ID":"2d1047f6-6bc7-422a-9633-04fcfb13920c","Type":"ContainerStarted","Data":"b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6"} Oct 31 00:21:01.460953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:01.460804 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" event={"ID":"2d1047f6-6bc7-422a-9633-04fcfb13920c","Type":"ContainerStarted","Data":"f59b8cf9d91792f5989e25773c93fe17efa29f788813f4e6e18182e10bb21852"} Oct 31 00:21:01.578440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:01.578409 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:01.593215 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:01.593141 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" podStartSLOduration=2.593118994 podStartE2EDuration="2.593118994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:01.585019375 +0000 UTC m=+1094.060377799" watchObservedRunningTime="2026-10-31 00:21:01.593118994 +0000 UTC m=+1094.068477419" Oct 31 00:21:02.601725 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:02.601688 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:21:02.602122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:02.601748 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:21:02.729854 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:21:02.729822 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:21:03.595630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:03.583999 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:21:03.595630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:03.584064 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:21:04.582027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:04.581991 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:21:04.582650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:04.582614 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:21:05.592002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:05.591962 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:21:05.592617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:05.592555 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:21:06.594617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:06.590943 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:21:06.594617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:06.591005 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:21:07.698721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:07.698686 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:07.699243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:07.699216 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:07.700158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:07.700139 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:21:16.831629 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:16.831578 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz"] Oct 31 00:21:16.835776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:16.835755 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:16.891425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:16.891402 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:21:16.891744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:16.891723 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:21:16.937143 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:16.937111 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce7e478-wz2lz\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:16.937318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:16.937155 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjlf\" (UniqueName: \"kubernetes.io/projected/c762c568-bcfa-4ec8-af87-c7fc1642075c-kube-api-access-xwjlf\") pod \"maas-api-revoke-sub-1ce7e478-wz2lz\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:16.937318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:16.937299 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce7e478-wz2lz\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:16.944807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:16.944778 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz"] Oct 31 00:21:17.038949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:17.038918 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce7e478-wz2lz\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:17.039119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:17.039050 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce7e478-wz2lz\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:17.039119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:17.039076 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjlf\" (UniqueName: \"kubernetes.io/projected/c762c568-bcfa-4ec8-af87-c7fc1642075c-kube-api-access-xwjlf\") pod \"maas-api-revoke-sub-1ce7e478-wz2lz\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:17.039832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:17.039806 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce7e478-wz2lz\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:17.041603 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:17.041568 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce7e478-wz2lz\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:17.103830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:17.103765 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjlf\" (UniqueName: \"kubernetes.io/projected/c762c568-bcfa-4ec8-af87-c7fc1642075c-kube-api-access-xwjlf\") pod \"maas-api-revoke-sub-1ce7e478-wz2lz\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:17.150767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:17.150736 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:17.534252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:17.534225 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz"] Oct 31 00:21:17.534953 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:21:17.534921 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc762c568_bcfa_4ec8_af87_c7fc1642075c.slice/crio-07856d499f23c3a8592e44a4f0c0b94da1761894fc4d1c899020adc9eba13b9a WatchSource:0}: Error finding container 07856d499f23c3a8592e44a4f0c0b94da1761894fc4d1c899020adc9eba13b9a: Status 404 returned error can't find the container with id 07856d499f23c3a8592e44a4f0c0b94da1761894fc4d1c899020adc9eba13b9a Oct 31 00:21:17.597914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:17.597876 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" event={"ID":"c762c568-bcfa-4ec8-af87-c7fc1642075c","Type":"ContainerStarted","Data":"07856d499f23c3a8592e44a4f0c0b94da1761894fc4d1c899020adc9eba13b9a"} Oct 31 00:21:18.605860 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:18.605819 4969 generic.go:356] "Generic (PLEG): container finished" podID="c762c568-bcfa-4ec8-af87-c7fc1642075c" containerID="3c000120b8a352c8a24aad0a2e45db9ebb5a42c78d1b5177cab21309c88e851e" exitCode=0 Oct 31 00:21:18.606263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:18.605874 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" event={"ID":"c762c568-bcfa-4ec8-af87-c7fc1642075c","Type":"ContainerDied","Data":"3c000120b8a352c8a24aad0a2e45db9ebb5a42c78d1b5177cab21309c88e851e"} Oct 31 00:21:19.761697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.761674 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:19.877230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.877204 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c762c568-bcfa-4ec8-af87-c7fc1642075c-kube-api-access-xwjlf\" (UniqueName: \"kubernetes.io/projected/c762c568-bcfa-4ec8-af87-c7fc1642075c-kube-api-access-xwjlf\") pod \"c762c568-bcfa-4ec8-af87-c7fc1642075c\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " Oct 31 00:21:19.877460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.877342 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-openshift-ca\") pod \"c762c568-bcfa-4ec8-af87-c7fc1642075c\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " Oct 31 00:21:19.877460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.877396 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-xks-ca\") pod \"c762c568-bcfa-4ec8-af87-c7fc1642075c\" (UID: \"c762c568-bcfa-4ec8-af87-c7fc1642075c\") " Oct 31 00:21:19.877831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.877805 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-openshift-ca" pod "c762c568-bcfa-4ec8-af87-c7fc1642075c" (UID: "c762c568-bcfa-4ec8-af87-c7fc1642075c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:19.880067 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.880040 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c762c568-bcfa-4ec8-af87-c7fc1642075c-kube-api-access-xwjlf" pod "c762c568-bcfa-4ec8-af87-c7fc1642075c" (UID: "c762c568-bcfa-4ec8-af87-c7fc1642075c"). InnerVolumeSpecName "kube-api-access-xwjlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:19.880551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.880515 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-xks-ca" pod "c762c568-bcfa-4ec8-af87-c7fc1642075c" (UID: "c762c568-bcfa-4ec8-af87-c7fc1642075c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:19.978614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.978559 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:19.978614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.978586 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjlf\" (UniqueName: \"kubernetes.io/projected/c762c568-bcfa-4ec8-af87-c7fc1642075c-kube-api-access-xwjlf\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:19.978802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:19.978623 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c762c568-bcfa-4ec8-af87-c7fc1642075c-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:20.620895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:20.620858 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" event={"ID":"c762c568-bcfa-4ec8-af87-c7fc1642075c","Type":"ContainerDied","Data":"07856d499f23c3a8592e44a4f0c0b94da1761894fc4d1c899020adc9eba13b9a"} Oct 31 00:21:20.621077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:20.620930 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07856d499f23c3a8592e44a4f0c0b94da1761894fc4d1c899020adc9eba13b9a" Oct 31 00:21:20.621077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:20.620956 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz" Oct 31 00:21:25.475546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.475510 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f"] Oct 31 00:21:25.476096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.476081 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c762c568-bcfa-4ec8-af87-c7fc1642075c" containerName="revoke-keys" Oct 31 00:21:25.476147 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.476098 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c762c568-bcfa-4ec8-af87-c7fc1642075c" containerName="revoke-keys" Oct 31 00:21:25.476192 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.476183 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c762c568-bcfa-4ec8-af87-c7fc1642075c" containerName="revoke-keys" Oct 31 00:21:25.479477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.479463 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.538026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.537999 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f"] Oct 31 00:21:25.540659 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.540621 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14116ff6-9f47f\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.540783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.540669 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14116ff6-9f47f\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.540844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.540814 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldfrc\" (UniqueName: \"kubernetes.io/projected/91ea92f5-9159-4a74-bfc2-275601c12cba-kube-api-access-ldfrc\") pod \"maas-api-revoke-sub-14116ff6-9f47f\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.582692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.582668 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:21:25.582867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.582850 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:21:25.641745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.641721 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14116ff6-9f47f\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.641863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.641751 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14116ff6-9f47f\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.641863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.641821 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldfrc\" (UniqueName: \"kubernetes.io/projected/91ea92f5-9159-4a74-bfc2-275601c12cba-kube-api-access-ldfrc\") pod \"maas-api-revoke-sub-14116ff6-9f47f\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.642328 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.642305 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14116ff6-9f47f\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.644230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.644210 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14116ff6-9f47f\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.703737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.703711 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldfrc\" (UniqueName: \"kubernetes.io/projected/91ea92f5-9159-4a74-bfc2-275601c12cba-kube-api-access-ldfrc\") pod \"maas-api-revoke-sub-14116ff6-9f47f\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:25.792792 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:25.792706 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:26.167845 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:26.167808 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f"] Oct 31 00:21:26.169175 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:21:26.169149 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ea92f5_9159_4a74_bfc2_275601c12cba.slice/crio-20dffe074191410ee18516dfa4ba22988f67cc00568e785f7f7c000e1ebc81ed WatchSource:0}: Error finding container 20dffe074191410ee18516dfa4ba22988f67cc00568e785f7f7c000e1ebc81ed: Status 404 returned error can't find the container with id 20dffe074191410ee18516dfa4ba22988f67cc00568e785f7f7c000e1ebc81ed Oct 31 00:21:26.662445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:26.662405 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" event={"ID":"91ea92f5-9159-4a74-bfc2-275601c12cba","Type":"ContainerStarted","Data":"8ffcffa3b96071cef1643774d0c7d16bd783f4f74e0e74796beae24f6903ea88"} Oct 31 00:21:26.662868 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:26.662498 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" event={"ID":"91ea92f5-9159-4a74-bfc2-275601c12cba","Type":"ContainerStarted","Data":"20dffe074191410ee18516dfa4ba22988f67cc00568e785f7f7c000e1ebc81ed"} Oct 31 00:21:26.782430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:26.782340 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" podStartSLOduration=1.7823154589999999 podStartE2EDuration="1.782315459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:26.781491505 +0000 UTC m=+1119.256849929" watchObservedRunningTime="2026-10-31 00:21:26.782315459 +0000 UTC m=+1119.257673881" Oct 31 00:21:27.671309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:27.671277 4969 generic.go:356] "Generic (PLEG): container finished" podID="91ea92f5-9159-4a74-bfc2-275601c12cba" containerID="8ffcffa3b96071cef1643774d0c7d16bd783f4f74e0e74796beae24f6903ea88" exitCode=0 Oct 31 00:21:27.671749 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:27.671328 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" event={"ID":"91ea92f5-9159-4a74-bfc2-275601c12cba","Type":"ContainerDied","Data":"8ffcffa3b96071cef1643774d0c7d16bd783f4f74e0e74796beae24f6903ea88"} Oct 31 00:21:28.830739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:28.830714 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:28.975888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:28.975816 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ea92f5-9159-4a74-bfc2-275601c12cba-kube-api-access-ldfrc\" (UniqueName: \"kubernetes.io/projected/91ea92f5-9159-4a74-bfc2-275601c12cba-kube-api-access-ldfrc\") pod \"91ea92f5-9159-4a74-bfc2-275601c12cba\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " Oct 31 00:21:28.975888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:28.975871 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-openshift-ca\") pod \"91ea92f5-9159-4a74-bfc2-275601c12cba\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " Oct 31 00:21:28.976075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:28.975907 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-xks-ca\") pod \"91ea92f5-9159-4a74-bfc2-275601c12cba\" (UID: \"91ea92f5-9159-4a74-bfc2-275601c12cba\") " Oct 31 00:21:28.976318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:28.976290 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-openshift-ca" pod "91ea92f5-9159-4a74-bfc2-275601c12cba" (UID: "91ea92f5-9159-4a74-bfc2-275601c12cba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:28.978211 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:28.978188 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-xks-ca" pod "91ea92f5-9159-4a74-bfc2-275601c12cba" (UID: "91ea92f5-9159-4a74-bfc2-275601c12cba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:28.978345 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:28.978323 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ea92f5-9159-4a74-bfc2-275601c12cba-kube-api-access-ldfrc" pod "91ea92f5-9159-4a74-bfc2-275601c12cba" (UID: "91ea92f5-9159-4a74-bfc2-275601c12cba"). InnerVolumeSpecName "kube-api-access-ldfrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:29.077752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.077722 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldfrc\" (UniqueName: \"kubernetes.io/projected/91ea92f5-9159-4a74-bfc2-275601c12cba-kube-api-access-ldfrc\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:29.077752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.077746 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:29.077752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.077756 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ea92f5-9159-4a74-bfc2-275601c12cba-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:29.335159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.335081 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc"] Oct 31 00:21:29.335586 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.335574 4969 cpu_manager.go:414] "containerMap: removing container" podUID="91ea92f5-9159-4a74-bfc2-275601c12cba" containerName="revoke-keys" Oct 31 00:21:29.335586 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.335611 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ea92f5-9159-4a74-bfc2-275601c12cba" containerName="revoke-keys" Oct 31 00:21:29.335726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.335716 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ea92f5-9159-4a74-bfc2-275601c12cba" containerName="revoke-keys" Oct 31 00:21:29.339315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.339292 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.398760 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.398726 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc"] Oct 31 00:21:29.482138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.482103 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblhr\" (UniqueName: \"kubernetes.io/projected/9e8bd090-269d-4be1-a34a-e975efac3d0a-kube-api-access-sblhr\") pod \"maas-api-revoke-sub-1da885d4-mx8tc\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.482313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.482161 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da885d4-mx8tc\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.482313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.482299 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da885d4-mx8tc\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.583896 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.583863 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da885d4-mx8tc\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.584085 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.583930 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da885d4-mx8tc\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.584085 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.583996 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblhr\" (UniqueName: \"kubernetes.io/projected/9e8bd090-269d-4be1-a34a-e975efac3d0a-kube-api-access-sblhr\") pod \"maas-api-revoke-sub-1da885d4-mx8tc\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.584563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.584537 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da885d4-mx8tc\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.586469 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.586425 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da885d4-mx8tc\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.645290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.645266 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblhr\" (UniqueName: \"kubernetes.io/projected/9e8bd090-269d-4be1-a34a-e975efac3d0a-kube-api-access-sblhr\") pod \"maas-api-revoke-sub-1da885d4-mx8tc\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.654123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.654098 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:29.688122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.688094 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" Oct 31 00:21:29.688271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.688134 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f" event={"ID":"91ea92f5-9159-4a74-bfc2-275601c12cba","Type":"ContainerDied","Data":"20dffe074191410ee18516dfa4ba22988f67cc00568e785f7f7c000e1ebc81ed"} Oct 31 00:21:29.688271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:29.688216 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20dffe074191410ee18516dfa4ba22988f67cc00568e785f7f7c000e1ebc81ed" Oct 31 00:21:30.034990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:30.034962 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc"] Oct 31 00:21:30.035945 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:21:30.035921 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8bd090_269d_4be1_a34a_e975efac3d0a.slice/crio-23e26d9634cbb268161d1f9862b8cca57e745734a790cd1de4b5698af87b7c5d WatchSource:0}: Error finding container 23e26d9634cbb268161d1f9862b8cca57e745734a790cd1de4b5698af87b7c5d: Status 404 returned error can't find the container with id 23e26d9634cbb268161d1f9862b8cca57e745734a790cd1de4b5698af87b7c5d Oct 31 00:21:30.696500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:30.696467 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" event={"ID":"9e8bd090-269d-4be1-a34a-e975efac3d0a","Type":"ContainerStarted","Data":"da305d8bd754994efe3b05753db426e31eaae55167b0b628a4fee0f67a227e2a"} Oct 31 00:21:30.696693 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:30.696523 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" event={"ID":"9e8bd090-269d-4be1-a34a-e975efac3d0a","Type":"ContainerStarted","Data":"23e26d9634cbb268161d1f9862b8cca57e745734a790cd1de4b5698af87b7c5d"} Oct 31 00:21:30.813759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:30.813705 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" podStartSLOduration=1.813690305 podStartE2EDuration="1.813690305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:30.811950692 +0000 UTC m=+1123.287309133" watchObservedRunningTime="2026-10-31 00:21:30.813690305 +0000 UTC m=+1123.289048728" Oct 31 00:21:31.704508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:31.704475 4969 generic.go:356] "Generic (PLEG): container finished" podID="9e8bd090-269d-4be1-a34a-e975efac3d0a" containerID="da305d8bd754994efe3b05753db426e31eaae55167b0b628a4fee0f67a227e2a" exitCode=0 Oct 31 00:21:31.704950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:31.704558 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" event={"ID":"9e8bd090-269d-4be1-a34a-e975efac3d0a","Type":"ContainerDied","Data":"da305d8bd754994efe3b05753db426e31eaae55167b0b628a4fee0f67a227e2a"} Oct 31 00:21:32.730454 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:21:32.730421 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:21:32.868315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:32.868291 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:33.022502 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.022422 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-openshift-ca\") pod \"9e8bd090-269d-4be1-a34a-e975efac3d0a\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " Oct 31 00:21:33.022665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.022620 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8bd090-269d-4be1-a34a-e975efac3d0a-kube-api-access-sblhr\" (UniqueName: \"kubernetes.io/projected/9e8bd090-269d-4be1-a34a-e975efac3d0a-kube-api-access-sblhr\") pod \"9e8bd090-269d-4be1-a34a-e975efac3d0a\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " Oct 31 00:21:33.022731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.022682 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-xks-ca\") pod \"9e8bd090-269d-4be1-a34a-e975efac3d0a\" (UID: \"9e8bd090-269d-4be1-a34a-e975efac3d0a\") " Oct 31 00:21:33.022828 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.022799 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-openshift-ca" pod "9e8bd090-269d-4be1-a34a-e975efac3d0a" (UID: "9e8bd090-269d-4be1-a34a-e975efac3d0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:33.023070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.023049 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:33.024929 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.024907 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-xks-ca" pod "9e8bd090-269d-4be1-a34a-e975efac3d0a" (UID: "9e8bd090-269d-4be1-a34a-e975efac3d0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:33.025031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.024990 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8bd090-269d-4be1-a34a-e975efac3d0a-kube-api-access-sblhr" pod "9e8bd090-269d-4be1-a34a-e975efac3d0a" (UID: "9e8bd090-269d-4be1-a34a-e975efac3d0a"). InnerVolumeSpecName "kube-api-access-sblhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:33.123680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.123653 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e8bd090-269d-4be1-a34a-e975efac3d0a-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:33.123680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.123679 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sblhr\" (UniqueName: \"kubernetes.io/projected/9e8bd090-269d-4be1-a34a-e975efac3d0a-kube-api-access-sblhr\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:33.719187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.719160 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" Oct 31 00:21:33.719187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.719166 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc" event={"ID":"9e8bd090-269d-4be1-a34a-e975efac3d0a","Type":"ContainerDied","Data":"23e26d9634cbb268161d1f9862b8cca57e745734a790cd1de4b5698af87b7c5d"} Oct 31 00:21:33.719383 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:33.719208 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23e26d9634cbb268161d1f9862b8cca57e745734a790cd1de4b5698af87b7c5d" Oct 31 00:21:40.128639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.128604 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm"] Oct 31 00:21:40.129139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.129126 4969 cpu_manager.go:414] "containerMap: removing container" podUID="9e8bd090-269d-4be1-a34a-e975efac3d0a" containerName="revoke-keys" Oct 31 00:21:40.129199 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.129140 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8bd090-269d-4be1-a34a-e975efac3d0a" containerName="revoke-keys" Oct 31 00:21:40.129238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.129230 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8bd090-269d-4be1-a34a-e975efac3d0a" containerName="revoke-keys" Oct 31 00:21:40.132588 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.132564 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.191682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.191657 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm"] Oct 31 00:21:40.236333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.236305 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:21:40.236481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.236390 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:21:40.294064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.294034 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmpm\" (UniqueName: \"kubernetes.io/projected/03e6cd26-86fe-4089-ba09-144095cf16fb-kube-api-access-tqmpm\") pod \"maas-api-revoke-sub-1b8fb89f-c7mtm\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.294204 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.294149 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b8fb89f-c7mtm\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.294204 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.294189 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b8fb89f-c7mtm\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.395861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.395780 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmpm\" (UniqueName: \"kubernetes.io/projected/03e6cd26-86fe-4089-ba09-144095cf16fb-kube-api-access-tqmpm\") pod \"maas-api-revoke-sub-1b8fb89f-c7mtm\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.396020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.395880 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b8fb89f-c7mtm\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.396020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.395907 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b8fb89f-c7mtm\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.396559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.396536 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b8fb89f-c7mtm\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.398496 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.398474 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b8fb89f-c7mtm\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.457084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.457057 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmpm\" (UniqueName: \"kubernetes.io/projected/03e6cd26-86fe-4089-ba09-144095cf16fb-kube-api-access-tqmpm\") pod \"maas-api-revoke-sub-1b8fb89f-c7mtm\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:40.746114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:40.746034 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:41.126919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:41.126894 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm"] Oct 31 00:21:41.127340 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:21:41.127320 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e6cd26_86fe_4089_ba09_144095cf16fb.slice/crio-05bbd4659a3060d23a7e1a548a9bb112e079938b182459535920a11d0f424b83 WatchSource:0}: Error finding container 05bbd4659a3060d23a7e1a548a9bb112e079938b182459535920a11d0f424b83: Status 404 returned error can't find the container with id 05bbd4659a3060d23a7e1a548a9bb112e079938b182459535920a11d0f424b83 Oct 31 00:21:41.781475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:41.781393 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" event={"ID":"03e6cd26-86fe-4089-ba09-144095cf16fb","Type":"ContainerStarted","Data":"07fd95d2214ce754178f2ded975d1ff8dffc08c42bfa4c8c73ff1efe031700ba"} Oct 31 00:21:41.781475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:41.781455 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" event={"ID":"03e6cd26-86fe-4089-ba09-144095cf16fb","Type":"ContainerStarted","Data":"05bbd4659a3060d23a7e1a548a9bb112e079938b182459535920a11d0f424b83"} Oct 31 00:21:41.898998 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:41.898948 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" podStartSLOduration=1.898933792 podStartE2EDuration="1.898933792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:41.897477779 +0000 UTC m=+1134.372836202" watchObservedRunningTime="2026-10-31 00:21:41.898933792 +0000 UTC m=+1134.374292252" Oct 31 00:21:42.789615 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:42.789560 4969 generic.go:356] "Generic (PLEG): container finished" podID="03e6cd26-86fe-4089-ba09-144095cf16fb" containerID="07fd95d2214ce754178f2ded975d1ff8dffc08c42bfa4c8c73ff1efe031700ba" exitCode=0 Oct 31 00:21:42.790000 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:42.789632 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" event={"ID":"03e6cd26-86fe-4089-ba09-144095cf16fb","Type":"ContainerDied","Data":"07fd95d2214ce754178f2ded975d1ff8dffc08c42bfa4c8c73ff1efe031700ba"} Oct 31 00:21:43.944324 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:43.944298 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:44.032640 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.032611 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-openshift-ca\") pod \"03e6cd26-86fe-4089-ba09-144095cf16fb\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " Oct 31 00:21:44.032768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.032701 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e6cd26-86fe-4089-ba09-144095cf16fb-kube-api-access-tqmpm\" (UniqueName: \"kubernetes.io/projected/03e6cd26-86fe-4089-ba09-144095cf16fb-kube-api-access-tqmpm\") pod \"03e6cd26-86fe-4089-ba09-144095cf16fb\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " Oct 31 00:21:44.032768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.032758 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-xks-ca\") pod \"03e6cd26-86fe-4089-ba09-144095cf16fb\" (UID: \"03e6cd26-86fe-4089-ba09-144095cf16fb\") " Oct 31 00:21:44.033039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.033011 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-openshift-ca" pod "03e6cd26-86fe-4089-ba09-144095cf16fb" (UID: "03e6cd26-86fe-4089-ba09-144095cf16fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:44.035013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.034988 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e6cd26-86fe-4089-ba09-144095cf16fb-kube-api-access-tqmpm" pod "03e6cd26-86fe-4089-ba09-144095cf16fb" (UID: "03e6cd26-86fe-4089-ba09-144095cf16fb"). InnerVolumeSpecName "kube-api-access-tqmpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:44.035013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.034988 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-xks-ca" pod "03e6cd26-86fe-4089-ba09-144095cf16fb" (UID: "03e6cd26-86fe-4089-ba09-144095cf16fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:21:44.134438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.134404 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqmpm\" (UniqueName: \"kubernetes.io/projected/03e6cd26-86fe-4089-ba09-144095cf16fb-kube-api-access-tqmpm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:44.134438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.134438 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:44.134707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.134452 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e6cd26-86fe-4089-ba09-144095cf16fb-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:21:44.804257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.804182 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" Oct 31 00:21:44.804257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.804214 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm" event={"ID":"03e6cd26-86fe-4089-ba09-144095cf16fb","Type":"ContainerDied","Data":"05bbd4659a3060d23a7e1a548a9bb112e079938b182459535920a11d0f424b83"} Oct 31 00:21:44.804441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:21:44.804261 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05bbd4659a3060d23a7e1a548a9bb112e079938b182459535920a11d0f424b83" Oct 31 00:22:02.727619 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:22:02.727556 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:22:11.820986 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.820941 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28"] Oct 31 00:22:11.822664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.822636 4969 cpu_manager.go:414] "containerMap: removing container" podUID="03e6cd26-86fe-4089-ba09-144095cf16fb" containerName="revoke-keys" Oct 31 00:22:11.822664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.822664 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e6cd26-86fe-4089-ba09-144095cf16fb" containerName="revoke-keys" Oct 31 00:22:11.823033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.823014 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e6cd26-86fe-4089-ba09-144095cf16fb" containerName="revoke-keys" Oct 31 00:22:11.827798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.827773 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:11.882956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.882921 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28"] Oct 31 00:22:11.928248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.928223 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:22:11.928248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.928242 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:22:11.931777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.931754 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba3f881-74b28\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:11.931909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.931879 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9m4\" (UniqueName: \"kubernetes.io/projected/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-kube-api-access-nf9m4\") pod \"maas-api-revoke-sub-eba3f881-74b28\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:11.932013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:11.931994 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba3f881-74b28\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:12.033317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:12.033281 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9m4\" (UniqueName: \"kubernetes.io/projected/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-kube-api-access-nf9m4\") pod \"maas-api-revoke-sub-eba3f881-74b28\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:12.033505 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:12.033371 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba3f881-74b28\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:12.033575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:12.033503 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba3f881-74b28\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:12.034110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:12.034084 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba3f881-74b28\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:12.035916 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:12.035889 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba3f881-74b28\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:12.094561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:12.094496 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9m4\" (UniqueName: \"kubernetes.io/projected/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-kube-api-access-nf9m4\") pod \"maas-api-revoke-sub-eba3f881-74b28\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:12.142230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:12.142203 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:12.521222 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:12.521195 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28"] Oct 31 00:22:12.522898 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:22:12.522864 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96c9c7c_db60_4aef_928b_a0723a6ee8bc.slice/crio-0490223acceca60bb239e54078cd50cf5172dd144c7b15480acaa8988ce62148 WatchSource:0}: Error finding container 0490223acceca60bb239e54078cd50cf5172dd144c7b15480acaa8988ce62148: Status 404 returned error can't find the container with id 0490223acceca60bb239e54078cd50cf5172dd144c7b15480acaa8988ce62148 Oct 31 00:22:13.015250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:13.015206 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" event={"ID":"f96c9c7c-db60-4aef-928b-a0723a6ee8bc","Type":"ContainerStarted","Data":"e2baebddd445f9aa8f876cce8b85f9a76978322542ba97c3eb4a0190447262e9"} Oct 31 00:22:13.015798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:13.015279 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" event={"ID":"f96c9c7c-db60-4aef-928b-a0723a6ee8bc","Type":"ContainerStarted","Data":"0490223acceca60bb239e54078cd50cf5172dd144c7b15480acaa8988ce62148"} Oct 31 00:22:13.135087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:13.135036 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" podStartSLOduration=2.135022208 podStartE2EDuration="2.135022208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:13.132271575 +0000 UTC m=+1165.607629999" watchObservedRunningTime="2026-10-31 00:22:13.135022208 +0000 UTC m=+1165.610380630" Oct 31 00:22:14.031684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.031645 4969 generic.go:356] "Generic (PLEG): container finished" podID="f96c9c7c-db60-4aef-928b-a0723a6ee8bc" containerID="e2baebddd445f9aa8f876cce8b85f9a76978322542ba97c3eb4a0190447262e9" exitCode=0 Oct 31 00:22:14.032195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.031875 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" event={"ID":"f96c9c7c-db60-4aef-928b-a0723a6ee8bc","Type":"ContainerDied","Data":"e2baebddd445f9aa8f876cce8b85f9a76978322542ba97c3eb4a0190447262e9"} Oct 31 00:22:14.522388 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.522345 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm"] Oct 31 00:22:14.527269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.527242 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.669935 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.669897 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.670104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.670034 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqnv\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-kube-api-access-7sqnv\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.670151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.670097 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.670151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.670141 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d42e79-429b-48a0-b53f-42a836067d2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.670250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.670230 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.670284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.670267 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d42e79-429b-48a0-b53f-42a836067d2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.670335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.670320 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.670415 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.670403 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.670482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.670444 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.751200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.751167 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm"] Oct 31 00:22:14.771537 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.771497 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.771708 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.771549 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.771708 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.771620 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.771708 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.771657 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.771883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.771732 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqnv\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-kube-api-access-7sqnv\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.771883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.771775 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.771883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.771812 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d42e79-429b-48a0-b53f-42a836067d2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.771883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.771878 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.772083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.771901 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d42e79-429b-48a0-b53f-42a836067d2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.772572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.772510 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.772572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.772523 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d42e79-429b-48a0-b53f-42a836067d2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.772794 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.772764 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.773071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.773046 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.773381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.773326 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.775020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.774998 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.775715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.775691 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d42e79-429b-48a0-b53f-42a836067d2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.843323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.842918 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqnv\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-kube-api-access-7sqnv\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.846218 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.846199 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-6bnvm\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:14.848177 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:14.848158 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:15.189239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.189201 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:15.282332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.282295 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-kube-api-access-nf9m4\" (UniqueName: \"kubernetes.io/projected/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-kube-api-access-nf9m4\") pod \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " Oct 31 00:22:15.282522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.282375 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-xks-ca\") pod \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " Oct 31 00:22:15.282522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.282473 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-openshift-ca\") pod \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\" (UID: \"f96c9c7c-db60-4aef-928b-a0723a6ee8bc\") " Oct 31 00:22:15.284248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.284193 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-openshift-ca" pod "f96c9c7c-db60-4aef-928b-a0723a6ee8bc" (UID: "f96c9c7c-db60-4aef-928b-a0723a6ee8bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:22:15.285323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.285280 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-kube-api-access-nf9m4" pod "f96c9c7c-db60-4aef-928b-a0723a6ee8bc" (UID: "f96c9c7c-db60-4aef-928b-a0723a6ee8bc"). InnerVolumeSpecName "kube-api-access-nf9m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:15.285544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.285510 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-xks-ca" pod "f96c9c7c-db60-4aef-928b-a0723a6ee8bc" (UID: "f96c9c7c-db60-4aef-928b-a0723a6ee8bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:22:15.323911 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:22:15.323858 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d42e79_429b_48a0_b53f_42a836067d2c.slice/crio-d56da74098cff113e609bf80639d59f6023a97f97ad6b14f9db6b195af4dd720 WatchSource:0}: Error finding container d56da74098cff113e609bf80639d59f6023a97f97ad6b14f9db6b195af4dd720: Status 404 returned error can't find the container with id d56da74098cff113e609bf80639d59f6023a97f97ad6b14f9db6b195af4dd720 Oct 31 00:22:15.326160 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.326128 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:22:15.326265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.326206 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:22:15.326265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.326251 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:22:15.361007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.360982 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm"] Oct 31 00:22:15.384511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.384482 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9m4\" (UniqueName: \"kubernetes.io/projected/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-kube-api-access-nf9m4\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:22:15.384641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.384514 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:22:15.384641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:15.384530 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96c9c7c-db60-4aef-928b-a0723a6ee8bc-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:22:16.048861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:16.048777 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" event={"ID":"61d42e79-429b-48a0-b53f-42a836067d2c","Type":"ContainerStarted","Data":"2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24"} Oct 31 00:22:16.048861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:16.048832 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" event={"ID":"61d42e79-429b-48a0-b53f-42a836067d2c","Type":"ContainerStarted","Data":"d56da74098cff113e609bf80639d59f6023a97f97ad6b14f9db6b195af4dd720"} Oct 31 00:22:16.050268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:16.050244 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" Oct 31 00:22:16.050370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:16.050246 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28" event={"ID":"f96c9c7c-db60-4aef-928b-a0723a6ee8bc","Type":"ContainerDied","Data":"0490223acceca60bb239e54078cd50cf5172dd144c7b15480acaa8988ce62148"} Oct 31 00:22:16.050370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:16.050358 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0490223acceca60bb239e54078cd50cf5172dd144c7b15480acaa8988ce62148" Oct 31 00:22:16.223559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:16.223509 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podStartSLOduration=2.223496302 podStartE2EDuration="2.223496302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:16.218912981 +0000 UTC m=+1168.694271404" watchObservedRunningTime="2026-10-31 00:22:16.223496302 +0000 UTC m=+1168.698854794" Oct 31 00:22:16.848663 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:16.848623 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:16.849843 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:16.849813 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6bnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:22:16.849901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:16.849861 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:22:17.848680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:17.848651 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6bnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:22:17.849069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:17.848701 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:22:18.848735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:18.848703 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6bnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:22:18.849220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:18.848753 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:22:19.270385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.270351 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68"] Oct 31 00:22:19.270994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.270973 4969 cpu_manager.go:414] "containerMap: removing container" podUID="f96c9c7c-db60-4aef-928b-a0723a6ee8bc" containerName="revoke-keys" Oct 31 00:22:19.270994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.270994 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96c9c7c-db60-4aef-928b-a0723a6ee8bc" containerName="revoke-keys" Oct 31 00:22:19.271174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.271109 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96c9c7c-db60-4aef-928b-a0723a6ee8bc" containerName="revoke-keys" Oct 31 00:22:19.274718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.274696 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.398619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.376706 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"e2e-conflict-a-d925972c-kserve-self-signed-certs\"" Oct 31 00:22:19.407625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.406185 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68"] Oct 31 00:22:19.440539 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.440490 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca34f89-54dd-401d-8007-d4ffe2951b73-tls-certs\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.440777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.440652 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-tmp-dir\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.440777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.440687 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-model-cache\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.440777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.440719 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-dshm\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.440988 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.440794 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-home\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.440988 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.440843 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrb5z\" (UniqueName: \"kubernetes.io/projected/0ca34f89-54dd-401d-8007-d4ffe2951b73-kube-api-access-nrb5z\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.541843 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.541761 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca34f89-54dd-401d-8007-d4ffe2951b73-tls-certs\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.542002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.541851 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-tmp-dir\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.542002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.541872 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-model-cache\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.542002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.541894 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-dshm\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.542002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.541928 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-home\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.542179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.542002 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrb5z\" (UniqueName: \"kubernetes.io/projected/0ca34f89-54dd-401d-8007-d4ffe2951b73-kube-api-access-nrb5z\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.542335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.542311 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-model-cache\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.542508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.542369 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-home\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.542580 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.542402 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-tmp-dir\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.544647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.544619 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca34f89-54dd-401d-8007-d4ffe2951b73-tls-certs\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.544861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.544832 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-dshm\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.604276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.604250 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrb5z\" (UniqueName: \"kubernetes.io/projected/0ca34f89-54dd-401d-8007-d4ffe2951b73-kube-api-access-nrb5z\") pod \"e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.608116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.608085 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:19.848694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.848614 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6bnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:22:19.848835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:19.848682 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:22:20.016284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.016254 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68"] Oct 31 00:22:20.016907 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:22:20.016868 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ca34f89_54dd_401d_8007_d4ffe2951b73.slice/crio-0c38c25dc22297809933f5f34844f9f35a495bcee7cd7dd28b65d0f77675bd25 WatchSource:0}: Error finding container 0c38c25dc22297809933f5f34844f9f35a495bcee7cd7dd28b65d0f77675bd25: Status 404 returned error can't find the container with id 0c38c25dc22297809933f5f34844f9f35a495bcee7cd7dd28b65d0f77675bd25 Oct 31 00:22:20.106034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.105940 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" event={"ID":"0ca34f89-54dd-401d-8007-d4ffe2951b73","Type":"ContainerStarted","Data":"fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91"} Oct 31 00:22:20.106034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.105984 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" event={"ID":"0ca34f89-54dd-401d-8007-d4ffe2951b73","Type":"ContainerStarted","Data":"0c38c25dc22297809933f5f34844f9f35a495bcee7cd7dd28b65d0f77675bd25"} Oct 31 00:22:20.198618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.198087 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q"] Oct 31 00:22:20.204032 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.204008 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.222576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.222535 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" podStartSLOduration=1.222520972 podStartE2EDuration="1.222520972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:20.221768797 +0000 UTC m=+1172.697127220" watchObservedRunningTime="2026-10-31 00:22:20.222520972 +0000 UTC m=+1172.697879397" Oct 31 00:22:20.241066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.241042 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q"] Oct 31 00:22:20.256612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.256571 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 00:22:20.256722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.256586 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 00:22:20.256722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.256653 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 00:22:20.256722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.256710 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 00:22:20.256854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.256728 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-h8pxf\"" Oct 31 00:22:20.256854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.256765 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-aa8816631626\"" Oct 31 00:22:20.352511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.352473 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kk8j\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-kube-api-access-5kk8j\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.352705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.352548 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-plugins-config-volume\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.352705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.352606 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-routing-overlay-volume\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.352780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.352705 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-provider-credentials\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.453774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.453732 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-plugins-config-volume\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.453957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.453816 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-routing-overlay-volume\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.453957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.453870 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-provider-credentials\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.454089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.454000 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kk8j\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-kube-api-access-5kk8j\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.454525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.454498 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-routing-overlay-volume\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.454809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.454584 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-plugins-config-volume\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.456427 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.456404 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-provider-credentials\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.515525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.515489 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kk8j\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-kube-api-access-5kk8j\") pod \"payload-processing-external-model-66fb6cc5c5-f2b6q\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.517370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.517351 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:20.608950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.608885 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:20.848789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.848687 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6bnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:22:20.848789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.848747 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:22:20.922657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:20.922626 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q"] Oct 31 00:22:20.923768 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:22:20.923741 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14599e23_3b4b_462e_ba5c_59bdc661f17d.slice/crio-72a82ea1ac998cc0cc94fd97a19fdc5a7c775cbd8719c6eceadcd6e64e8c3bc6 WatchSource:0}: Error finding container 72a82ea1ac998cc0cc94fd97a19fdc5a7c775cbd8719c6eceadcd6e64e8c3bc6: Status 404 returned error can't find the container with id 72a82ea1ac998cc0cc94fd97a19fdc5a7c775cbd8719c6eceadcd6e64e8c3bc6 Oct 31 00:22:21.114992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:21.114908 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" event={"ID":"14599e23-3b4b-462e-ba5c-59bdc661f17d","Type":"ContainerStarted","Data":"1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b"} Oct 31 00:22:21.114992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:21.114965 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:22:21.114992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:21.114982 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" event={"ID":"14599e23-3b4b-462e-ba5c-59bdc661f17d","Type":"ContainerStarted","Data":"72a82ea1ac998cc0cc94fd97a19fdc5a7c775cbd8719c6eceadcd6e64e8c3bc6"} Oct 31 00:22:21.233976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:21.233920 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" podStartSLOduration=1.233906795 podStartE2EDuration="1.233906795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:21.23058028 +0000 UTC m=+1173.705938706" watchObservedRunningTime="2026-10-31 00:22:21.233906795 +0000 UTC m=+1173.709265218" Oct 31 00:22:21.848953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:21.848871 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6bnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:22:21.849402 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:21.848974 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:22:22.848995 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:22.848952 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6bnvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:22:22.849444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:22.849008 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:22:24.034566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:24.034537 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:24.035126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:24.035101 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:24.036212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:24.036185 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:22:29.609064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:29.609025 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:29.622431 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:29.622408 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:30.217398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:30.217374 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:22:32.135124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:32.135082 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:22:32.730436 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:22:32.730211 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:22:42.132998 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:42.132963 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:22:45.834897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.834866 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd"] Oct 31 00:22:45.839070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.839048 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:45.942711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.942676 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:45.942865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.942741 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:45.942927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.942861 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:45.942927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.942866 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd"] Oct 31 00:22:45.942927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.942921 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:45.943094 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.943074 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:45.943134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.943109 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:45.943206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.943184 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhd9\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-kube-api-access-gvhd9\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:45.943282 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.943235 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:45.943336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:45.943296 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044061 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044104 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044129 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhd9\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-kube-api-access-gvhd9\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044163 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044220 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044264 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044308 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044364 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044408 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044504 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044620 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044773 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.044856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.044809 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.045133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.045110 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.046739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.046713 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.046848 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.046813 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.105276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.105212 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.105466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.105443 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhd9\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-kube-api-access-gvhd9\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.152451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.152410 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:46.585931 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:22:46.585901 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ce29c6_7ec2_4b1f_a432_114f56ee0830.slice/crio-b2217240253d0b73c0464cb114cd268923b3af45d7bef50733720436fbbda8c0 WatchSource:0}: Error finding container b2217240253d0b73c0464cb114cd268923b3af45d7bef50733720436fbbda8c0: Status 404 returned error can't find the container with id b2217240253d0b73c0464cb114cd268923b3af45d7bef50733720436fbbda8c0 Oct 31 00:22:46.585931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.585909 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd"] Oct 31 00:22:46.588434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.588400 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:22:46.588540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.588480 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:22:46.588540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:46.588521 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:22:47.335899 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:47.335855 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" event={"ID":"a3ce29c6-7ec2-4b1f-a432-114f56ee0830","Type":"ContainerStarted","Data":"b97bde132c97e01aac4f0677e0ccec9a2643cdd5d33851b63dab8722f47e8f89"} Oct 31 00:22:47.336290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:47.335951 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" event={"ID":"a3ce29c6-7ec2-4b1f-a432-114f56ee0830","Type":"ContainerStarted","Data":"b2217240253d0b73c0464cb114cd268923b3af45d7bef50733720436fbbda8c0"} Oct 31 00:22:47.560911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:47.560843 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" podStartSLOduration=2.56082993 podStartE2EDuration="2.56082993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:47.506441543 +0000 UTC m=+1199.981799965" watchObservedRunningTime="2026-10-31 00:22:47.56082993 +0000 UTC m=+1200.036188422" Oct 31 00:22:47.869801 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:47.869770 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:22:47.871765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:47.871740 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:22:47.911922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:47.911897 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:22:47.913534 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:47.913517 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:22:48.153458 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:48.153367 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:48.154987 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:48.154963 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:48.155066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:48.155015 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:49.153092 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:49.153055 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:49.153482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:49.153109 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:50.153664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:50.153624 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:50.154068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:50.153692 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:51.152753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:51.152719 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:51.152941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:51.152779 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:52.133769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:52.133731 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:22:52.153138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:52.153109 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:52.153249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:52.153158 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:53.153235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:53.153204 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:53.153676 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:53.153261 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:54.156642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:54.156616 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:54.157034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:54.157014 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:22:54.157645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:22:54.157621 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:23:02.133239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:02.133147 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:23:02.727524 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:23:02.727488 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:23:12.133082 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:12.133049 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:23:16.573459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.573428 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bnwts-jlqxx"] Oct 31 00:23:16.577554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.577533 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.650898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.650870 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hkxnt\"" Oct 31 00:23:16.704321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.704293 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bnwts-jlqxx"] Oct 31 00:23:16.706187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.706161 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvsr8\" (UniqueName: \"kubernetes.io/projected/58101b4a-2aef-45e9-b534-89351404d96e-kube-api-access-lvsr8\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.706349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.706226 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-service-ca-bundle\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.706446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.706428 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-trusted-ca-bundle\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.706515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.706484 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58101b4a-2aef-45e9-b534-89351404d96e-archives-path\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.807349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.807314 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-service-ca-bundle\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.807550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.807389 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-trusted-ca-bundle\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.807550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.807415 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58101b4a-2aef-45e9-b534-89351404d96e-archives-path\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.807550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.807494 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvsr8\" (UniqueName: \"kubernetes.io/projected/58101b4a-2aef-45e9-b534-89351404d96e-kube-api-access-lvsr8\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.807937 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.807912 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58101b4a-2aef-45e9-b534-89351404d96e-archives-path\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.808123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.808100 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-service-ca-bundle\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.808186 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.808149 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-trusted-ca-bundle\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.870807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.870784 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvsr8\" (UniqueName: \"kubernetes.io/projected/58101b4a-2aef-45e9-b534-89351404d96e-kube-api-access-lvsr8\") pod \"periodic-gathering-bnwts-jlqxx\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:16.889546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:16.889522 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:23:17.271182 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:17.271152 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bnwts-jlqxx"] Oct 31 00:23:17.271725 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:23:17.271695 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58101b4a_2aef_45e9_b534_89351404d96e.slice/crio-722a720b670e712e0d550516e2b1d731d5f4c2d09760e3b876722bd4c037abf2 WatchSource:0}: Error finding container 722a720b670e712e0d550516e2b1d731d5f4c2d09760e3b876722bd4c037abf2: Status 404 returned error can't find the container with id 722a720b670e712e0d550516e2b1d731d5f4c2d09760e3b876722bd4c037abf2 Oct 31 00:23:17.553513 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:17.553422 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bnwts-jlqxx" event={"ID":"58101b4a-2aef-45e9-b534-89351404d96e","Type":"ContainerStarted","Data":"6d86b158f7cfce17d284645ce3cd74e7097ee1c3f86489d0507fb842dbc5215f"} Oct 31 00:23:17.553513 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:17.553480 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bnwts-jlqxx" event={"ID":"58101b4a-2aef-45e9-b534-89351404d96e","Type":"ContainerStarted","Data":"722a720b670e712e0d550516e2b1d731d5f4c2d09760e3b876722bd4c037abf2"} Oct 31 00:23:17.668071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:17.668016 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bnwts-jlqxx" podStartSLOduration=1.668002542 podStartE2EDuration="1.668002542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:17.667390589 +0000 UTC m=+1230.142749015" watchObservedRunningTime="2026-10-31 00:23:17.668002542 +0000 UTC m=+1230.143360965" Oct 31 00:23:22.133327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:22.133285 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:23:23.284762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.284731 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q"] Oct 31 00:23:23.286182 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.286110 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:23:23.319714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.319685 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n"] Oct 31 00:23:23.323902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.323880 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.376045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.376022 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a2f1ffbe7a37\"" Oct 31 00:23:23.396773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.396744 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n"] Oct 31 00:23:23.485561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.485528 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmwl\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-kube-api-access-6jmwl\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.485561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.485563 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-provider-credentials\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.485789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.485585 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-plugins-config-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.485845 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.485831 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-routing-overlay-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.587554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.587479 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-routing-overlay-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.587554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.587531 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmwl\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-kube-api-access-6jmwl\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.587554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.587550 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-provider-credentials\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.587893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.587579 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-plugins-config-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.588177 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.588155 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-routing-overlay-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.588277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.588258 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-plugins-config-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.590141 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.590119 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-provider-credentials\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.648429 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.648397 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmwl\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-kube-api-access-6jmwl\") pod \"payload-processing-external-model-7bcdc9f9cc-wnf8n\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:23.937134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:23.937105 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:24.317420 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:24.317395 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n"] Oct 31 00:23:24.317880 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:23:24.317715 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194e388d_96b0_4a1c_9124_3daccc692b4f.slice/crio-28ff3d36fb92dc1d97534e6502d7e4bac735b3ad1242d4f1e1db3ae48f108139 WatchSource:0}: Error finding container 28ff3d36fb92dc1d97534e6502d7e4bac735b3ad1242d4f1e1db3ae48f108139: Status 404 returned error can't find the container with id 28ff3d36fb92dc1d97534e6502d7e4bac735b3ad1242d4f1e1db3ae48f108139 Oct 31 00:23:24.611614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:24.611514 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" event={"ID":"194e388d-96b0-4a1c-9124-3daccc692b4f","Type":"ContainerStarted","Data":"f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225"} Oct 31 00:23:24.611614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:24.611563 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" event={"ID":"194e388d-96b0-4a1c-9124-3daccc692b4f","Type":"ContainerStarted","Data":"28ff3d36fb92dc1d97534e6502d7e4bac735b3ad1242d4f1e1db3ae48f108139"} Oct 31 00:23:24.611789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:24.611614 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:23:24.731388 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:24.731336 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podStartSLOduration=1.731321984 podStartE2EDuration="1.731321984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:24.727209468 +0000 UTC m=+1237.202567891" watchObservedRunningTime="2026-10-31 00:23:24.731321984 +0000 UTC m=+1237.206680406" Oct 31 00:23:28.286007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.285967 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" containerID="cri-o://1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b" gracePeriod=25 Oct 31 00:23:28.553919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.553860 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:23:28.640492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.640459 4969 generic.go:356] "Generic (PLEG): container finished" podID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerID="1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b" exitCode=0 Oct 31 00:23:28.640646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.640501 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" Oct 31 00:23:28.640646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.640551 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" event={"ID":"14599e23-3b4b-462e-ba5c-59bdc661f17d","Type":"ContainerDied","Data":"1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b"} Oct 31 00:23:28.640646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.640623 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q" event={"ID":"14599e23-3b4b-462e-ba5c-59bdc661f17d","Type":"ContainerDied","Data":"72a82ea1ac998cc0cc94fd97a19fdc5a7c775cbd8719c6eceadcd6e64e8c3bc6"} Oct 31 00:23:28.640771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.640652 4969 scope.go:122] "RemoveContainer" containerID="1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b" Oct 31 00:23:28.646039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.646016 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-routing-overlay-volume\") pod \"14599e23-3b4b-462e-ba5c-59bdc661f17d\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " Oct 31 00:23:28.646143 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.646078 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-kube-api-access-5kk8j\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-kube-api-access-5kk8j\") pod \"14599e23-3b4b-462e-ba5c-59bdc661f17d\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " Oct 31 00:23:28.646143 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.646121 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-plugins-config-volume\") pod \"14599e23-3b4b-462e-ba5c-59bdc661f17d\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " Oct 31 00:23:28.646265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.646199 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-provider-credentials\") pod \"14599e23-3b4b-462e-ba5c-59bdc661f17d\" (UID: \"14599e23-3b4b-462e-ba5c-59bdc661f17d\") " Oct 31 00:23:28.646388 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.646354 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-routing-overlay-volume" pod "14599e23-3b4b-462e-ba5c-59bdc661f17d" (UID: "14599e23-3b4b-462e-ba5c-59bdc661f17d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:28.646548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.646529 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-plugins-config-volume" pod "14599e23-3b4b-462e-ba5c-59bdc661f17d" (UID: "14599e23-3b4b-462e-ba5c-59bdc661f17d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:28.646638 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.646619 4969 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-routing-overlay-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:23:28.648441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.648413 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-kube-api-access-5kk8j" pod "14599e23-3b4b-462e-ba5c-59bdc661f17d" (UID: "14599e23-3b4b-462e-ba5c-59bdc661f17d"). InnerVolumeSpecName "kube-api-access-5kk8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:28.648543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.648433 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-provider-credentials" pod "14599e23-3b4b-462e-ba5c-59bdc661f17d" (UID: "14599e23-3b4b-462e-ba5c-59bdc661f17d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:28.651331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.651313 4969 scope.go:122] "RemoveContainer" containerID="1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b" Oct 31 00:23:28.651625 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:23:28.651585 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b\": container with ID starting with 1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b not found: ID does not exist" containerID="1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b" Oct 31 00:23:28.651699 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.651631 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b"} err="failed to get container status \"1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b\": rpc error: code = NotFound desc = could not find container \"1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b\": container with ID starting with 1bacfcb37bd117033a1b60dee1ccb34d013ae823131baf034473c7ab6a2f507b not found: ID does not exist" Oct 31 00:23:28.747418 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.747389 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14599e23-3b4b-462e-ba5c-59bdc661f17d-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:23:28.747418 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.747417 4969 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-provider-credentials\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:23:28.747624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:28.747430 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kk8j\" (UniqueName: \"kubernetes.io/projected/14599e23-3b4b-462e-ba5c-59bdc661f17d-kube-api-access-5kk8j\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:23:29.061969 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.061929 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q"] Oct 31 00:23:29.111640 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.111611 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-66fb6cc5c5-f2b6q"] Oct 31 00:23:29.520820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.520787 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt"] Oct 31 00:23:29.521365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.521350 4969 cpu_manager.go:414] "containerMap: removing container" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" Oct 31 00:23:29.521415 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.521367 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" Oct 31 00:23:29.521475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.521465 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" containerName="payload-processing" Oct 31 00:23:29.526099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.526073 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.556603 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.556552 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tmp-dir\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.556777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.556681 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9795h\" (UniqueName: \"kubernetes.io/projected/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-kube-api-access-9795h\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.556777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.556756 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-home\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.556958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.556940 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-model-cache\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.557027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.556978 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tls-certs\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.557080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.557059 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-dshm\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.583501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.583474 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt"] Oct 31 00:23:29.590818 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.590791 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"e2e-conflict-b-d925972c-kserve-self-signed-certs\"" Oct 31 00:23:29.658220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.658178 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-model-cache\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.660244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.660204 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tls-certs\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.660409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.660281 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-dshm\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.660409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.660359 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tmp-dir\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.660409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.660388 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9795h\" (UniqueName: \"kubernetes.io/projected/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-kube-api-access-9795h\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.660946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.660452 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-home\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.660946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.660916 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-model-cache\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.678208 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.663500 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-home\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.678208 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.663437 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tmp-dir\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.678208 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.664144 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-dshm\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.678208 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.665833 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tls-certs\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.724058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.724022 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9795h\" (UniqueName: \"kubernetes.io/projected/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-kube-api-access-9795h\") pod \"e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:29.735255 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.735227 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14599e23-3b4b-462e-ba5c-59bdc661f17d" path="/var/lib/kubelet/pods/14599e23-3b4b-462e-ba5c-59bdc661f17d/volumes" Oct 31 00:23:29.838735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:29.838660 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:30.227214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:30.227186 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt"] Oct 31 00:23:30.228027 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:23:30.228003 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb92eaeda_fb10_41fc_afcc_b7b53b74e2d2.slice/crio-6833f132409b7d5fb9a790413416dbbec446b97f0c59e22c919d6326b19827c4 WatchSource:0}: Error finding container 6833f132409b7d5fb9a790413416dbbec446b97f0c59e22c919d6326b19827c4: Status 404 returned error can't find the container with id 6833f132409b7d5fb9a790413416dbbec446b97f0c59e22c919d6326b19827c4 Oct 31 00:23:30.658892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:30.658857 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" event={"ID":"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2","Type":"ContainerStarted","Data":"b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6"} Oct 31 00:23:30.659342 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:30.658909 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" event={"ID":"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2","Type":"ContainerStarted","Data":"6833f132409b7d5fb9a790413416dbbec446b97f0c59e22c919d6326b19827c4"} Oct 31 00:23:30.778189 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:30.778144 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" podStartSLOduration=1.778131049 podStartE2EDuration="1.778131049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:30.774533019 +0000 UTC m=+1243.249891442" watchObservedRunningTime="2026-10-31 00:23:30.778131049 +0000 UTC m=+1243.253489471" Oct 31 00:23:30.839250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:30.839210 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:32.727553 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:23:32.727513 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:23:35.622566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:35.622527 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:23:39.839362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:39.839327 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:39.852699 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:39.852664 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:40.759652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:40.759622 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:23:45.621312 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:45.621256 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:23:47.977533 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:47.977503 4969 scope.go:122] "RemoveContainer" containerID="7215369b1936776748dcddd47e255ef78b061f81c91a6475797dddb1e6ab7f23" Oct 31 00:23:47.989021 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:47.988999 4969 scope.go:122] "RemoveContainer" containerID="375c5e1e113007ba7fe4ecdbafdbe433513b33792d55c0d19a1592edaacdfa7b" Oct 31 00:23:47.999323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:47.999301 4969 scope.go:122] "RemoveContainer" containerID="0b2a4e7acedce1f687d818695dd71a4ccfdc28e207986b492fcaad8f4ad8521a" Oct 31 00:23:55.620279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:23:55.620236 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:24:02.728031 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:24:02.727977 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:24:05.620147 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:05.620108 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:24:07.958812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:07.958775 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x"] Oct 31 00:24:07.962714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:07.962699 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.036144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.036110 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x"] Oct 31 00:24:08.066133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.066110 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e1ca71\"/\"distinct-2-w2-e1ca71-kserve-self-signed-certs\"" Oct 31 00:24:08.070230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.070199 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-home\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.070351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.070255 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-tmp-dir\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.070394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.070359 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-dshm\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.070432 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.070400 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-model-cache\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.070432 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.070423 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttjsm\" (UniqueName: \"kubernetes.io/projected/8a680cdc-06a0-432d-9684-c642d32c7fd3-kube-api-access-ttjsm\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.070575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.070560 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a680cdc-06a0-432d-9684-c642d32c7fd3-tls-certs\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.171355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.171319 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-home\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.171528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.171386 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-tmp-dir\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.171528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.171441 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-dshm\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.171528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.171480 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-model-cache\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.171528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.171509 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttjsm\" (UniqueName: \"kubernetes.io/projected/8a680cdc-06a0-432d-9684-c642d32c7fd3-kube-api-access-ttjsm\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.171734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.171663 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a680cdc-06a0-432d-9684-c642d32c7fd3-tls-certs\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.171841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.171816 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-tmp-dir\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.171907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.171828 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-home\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.171907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.171879 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-model-cache\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.174030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.174001 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-dshm\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.174243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.174225 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a680cdc-06a0-432d-9684-c642d32c7fd3-tls-certs\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.235907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.235834 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttjsm\" (UniqueName: \"kubernetes.io/projected/8a680cdc-06a0-432d-9684-c642d32c7fd3-kube-api-access-ttjsm\") pod \"distinct-2-w2-e1ca71-kserve-6f958854-w2m8x\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.275298 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.275271 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:08.680027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.679999 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x"] Oct 31 00:24:08.680173 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:24:08.680110 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a680cdc_06a0_432d_9684_c642d32c7fd3.slice/crio-b245762519a34acbd179160b6b309105babb00e7568f01c3e9393c7681da026b WatchSource:0}: Error finding container b245762519a34acbd179160b6b309105babb00e7568f01c3e9393c7681da026b: Status 404 returned error can't find the container with id b245762519a34acbd179160b6b309105babb00e7568f01c3e9393c7681da026b Oct 31 00:24:08.991210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.991120 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" event={"ID":"8a680cdc-06a0-432d-9684-c642d32c7fd3","Type":"ContainerStarted","Data":"ac94b455dec9d0b4c2c9d1b0969dccacd978049b7421ff01b2e68464d4fec123"} Oct 31 00:24:08.991210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:08.991169 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" event={"ID":"8a680cdc-06a0-432d-9684-c642d32c7fd3","Type":"ContainerStarted","Data":"b245762519a34acbd179160b6b309105babb00e7568f01c3e9393c7681da026b"} Oct 31 00:24:09.111328 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:09.111274 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podStartSLOduration=2.111256111 podStartE2EDuration="2.111256111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:09.110255353 +0000 UTC m=+1281.585613776" watchObservedRunningTime="2026-10-31 00:24:09.111256111 +0000 UTC m=+1281.586614538" Oct 31 00:24:09.276371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:09.276290 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:15.620503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:15.620462 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:24:18.275930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:18.275891 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:18.295764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:18.295734 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:19.094690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:19.094661 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:24:21.703930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.703892 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s"] Oct 31 00:24:21.708434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.708408 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.765800 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.765765 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s"] Oct 31 00:24:21.810944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.810910 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"openshift-service-ca.crt\"" Oct 31 00:24:21.811110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.810976 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"default-dockercfg-g9zff\"" Oct 31 00:24:21.811110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.810993 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"kube-root-ca.crt\"" Oct 31 00:24:21.811110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.811052 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"distinct-w4-0144bd-kserve-self-signed-certs\"" Oct 31 00:24:21.835877 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.835841 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tls-certs\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.836049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.835880 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-dshm\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.836126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.836098 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-home\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.836183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.836170 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tmp-dir\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.836236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.836199 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrq7\" (UniqueName: \"kubernetes.io/projected/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-kube-api-access-wmrq7\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.836334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.836312 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-model-cache\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.937267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.937229 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tmp-dir\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.937267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.937270 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrq7\" (UniqueName: \"kubernetes.io/projected/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-kube-api-access-wmrq7\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.937535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.937367 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-model-cache\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.937535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.937401 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tls-certs\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.937535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.937430 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-dshm\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.937734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.937561 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-home\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.937797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.937747 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tmp-dir\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.937797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.937765 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-model-cache\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.937963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.937944 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-home\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.940074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.940049 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-dshm\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.940257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.940241 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tls-certs\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:21.998313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:21.998251 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrq7\" (UniqueName: \"kubernetes.io/projected/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-kube-api-access-wmrq7\") pod \"distinct-w4-0144bd-kserve-668949c45b-rlq8s\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:22.022121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:22.022096 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:22.438241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:22.438216 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s"] Oct 31 00:24:22.439249 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:24:22.439222 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda618e25b_20c7_4424_a7ac_3d5a89d9a24a.slice/crio-eb687b9ae688240937eba8b289b1f3ba8d0d38efec40c0801992bcce643a660f WatchSource:0}: Error finding container eb687b9ae688240937eba8b289b1f3ba8d0d38efec40c0801992bcce643a660f: Status 404 returned error can't find the container with id eb687b9ae688240937eba8b289b1f3ba8d0d38efec40c0801992bcce643a660f Oct 31 00:24:23.153069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:23.153027 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" event={"ID":"a618e25b-20c7-4424-a7ac-3d5a89d9a24a","Type":"ContainerStarted","Data":"6c6048760df7f81d9a2f5baa9e58449d6676018b7f93eb62e07436faa3a20716"} Oct 31 00:24:23.153069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:23.153070 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" event={"ID":"a618e25b-20c7-4424-a7ac-3d5a89d9a24a","Type":"ContainerStarted","Data":"eb687b9ae688240937eba8b289b1f3ba8d0d38efec40c0801992bcce643a660f"} Oct 31 00:24:23.270616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:23.270540 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podStartSLOduration=2.270527198 podStartE2EDuration="2.270527198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:23.26894415 +0000 UTC m=+1295.744302574" watchObservedRunningTime="2026-10-31 00:24:23.270527198 +0000 UTC m=+1295.745885620" Oct 31 00:24:24.022510 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:24.022478 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:25.620706 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:25.620673 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:24:32.022453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:32.022414 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:32.034911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:32.034886 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:32.244868 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:32.244840 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:24:32.726466 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:24:32.726432 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:24:35.620683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:35.620643 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:24:41.444022 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.443992 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n"] Oct 31 00:24:41.445350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.445318 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:24:41.487934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.487905 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6"] Oct 31 00:24:41.492041 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.492024 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.545057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.545035 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-43bfee711d1b\"" Oct 31 00:24:41.545309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.545061 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-282885dac35a\"" Oct 31 00:24:41.549554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.549532 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6"] Oct 31 00:24:41.562516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.562477 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-routing-overlay-volume\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.562695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.562541 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-plugins-config-volume\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.562767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.562696 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b69bf\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-kube-api-access-b69bf\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.562767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.562738 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.663973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.663940 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-routing-overlay-volume\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.663973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.663978 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-plugins-config-volume\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.664220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.664028 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b69bf\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-kube-api-access-b69bf\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.664220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.664051 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.664688 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.664662 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-routing-overlay-volume\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.664847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.664825 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-plugins-config-volume\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.666833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.666816 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.725433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.725367 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69bf\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-kube-api-access-b69bf\") pod \"payload-processing-external-model-657c7b7984-zkcf6\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:41.806354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:41.806327 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:42.187196 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:24:42.187164 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf598e6e3_f02b_4750_b717_ff5ecba6477c.slice/crio-684497b76a2e8719c3152fa6feb94baedf8370e6a553d5db58bb35669d5d0e8c WatchSource:0}: Error finding container 684497b76a2e8719c3152fa6feb94baedf8370e6a553d5db58bb35669d5d0e8c: Status 404 returned error can't find the container with id 684497b76a2e8719c3152fa6feb94baedf8370e6a553d5db58bb35669d5d0e8c Oct 31 00:24:42.187317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:42.187286 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6"] Oct 31 00:24:42.309482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:42.309447 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" event={"ID":"f598e6e3-f02b-4750-b717-ff5ecba6477c","Type":"ContainerStarted","Data":"f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7"} Oct 31 00:24:42.309709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:42.309527 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" event={"ID":"f598e6e3-f02b-4750-b717-ff5ecba6477c","Type":"ContainerStarted","Data":"684497b76a2e8719c3152fa6feb94baedf8370e6a553d5db58bb35669d5d0e8c"} Oct 31 00:24:42.309709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:42.309547 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:42.427711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:42.427664 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" podStartSLOduration=1.427649269 podStartE2EDuration="1.427649269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:42.425973595 +0000 UTC m=+1314.901332017" watchObservedRunningTime="2026-10-31 00:24:42.427649269 +0000 UTC m=+1314.903007692" Oct 31 00:24:42.607281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:42.607195 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt"] Oct 31 00:24:46.444684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.444638 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" containerID="cri-o://f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225" gracePeriod=25 Oct 31 00:24:46.712620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.712544 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:24:46.827704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.827672 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-routing-overlay-volume\") pod \"194e388d-96b0-4a1c-9124-3daccc692b4f\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " Oct 31 00:24:46.827862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.827719 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-provider-credentials\") pod \"194e388d-96b0-4a1c-9124-3daccc692b4f\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " Oct 31 00:24:46.827862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.827798 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-plugins-config-volume\") pod \"194e388d-96b0-4a1c-9124-3daccc692b4f\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " Oct 31 00:24:46.827862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.827842 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-kube-api-access-6jmwl\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-kube-api-access-6jmwl\") pod \"194e388d-96b0-4a1c-9124-3daccc692b4f\" (UID: \"194e388d-96b0-4a1c-9124-3daccc692b4f\") " Oct 31 00:24:46.828018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.827998 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-routing-overlay-volume" pod "194e388d-96b0-4a1c-9124-3daccc692b4f" (UID: "194e388d-96b0-4a1c-9124-3daccc692b4f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:46.828277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.828254 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-plugins-config-volume" pod "194e388d-96b0-4a1c-9124-3daccc692b4f" (UID: "194e388d-96b0-4a1c-9124-3daccc692b4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:46.828347 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.828328 4969 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-routing-overlay-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:46.830841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.830802 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-provider-credentials" pod "194e388d-96b0-4a1c-9124-3daccc692b4f" (UID: "194e388d-96b0-4a1c-9124-3daccc692b4f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:46.835777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.835747 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-kube-api-access-6jmwl" pod "194e388d-96b0-4a1c-9124-3daccc692b4f" (UID: "194e388d-96b0-4a1c-9124-3daccc692b4f"). InnerVolumeSpecName "kube-api-access-6jmwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:46.931417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.931387 4969 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-provider-credentials\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:46.931417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.931411 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194e388d-96b0-4a1c-9124-3daccc692b4f-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:46.931417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:46.931422 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jmwl\" (UniqueName: \"kubernetes.io/projected/194e388d-96b0-4a1c-9124-3daccc692b4f-kube-api-access-6jmwl\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:47.344326 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.344288 4969 generic.go:356] "Generic (PLEG): container finished" podID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerID="f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225" exitCode=0 Oct 31 00:24:47.344473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.344356 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" Oct 31 00:24:47.344473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.344372 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" event={"ID":"194e388d-96b0-4a1c-9124-3daccc692b4f","Type":"ContainerDied","Data":"f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225"} Oct 31 00:24:47.344473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.344454 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n" event={"ID":"194e388d-96b0-4a1c-9124-3daccc692b4f","Type":"ContainerDied","Data":"28ff3d36fb92dc1d97534e6502d7e4bac735b3ad1242d4f1e1db3ae48f108139"} Oct 31 00:24:47.344624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.344475 4969 scope.go:122] "RemoveContainer" containerID="f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225" Oct 31 00:24:47.355335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.355312 4969 scope.go:122] "RemoveContainer" containerID="f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225" Oct 31 00:24:47.355648 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:24:47.355620 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225\": container with ID starting with f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225 not found: ID does not exist" containerID="f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225" Oct 31 00:24:47.355718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.355661 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225"} err="failed to get container status \"f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225\": rpc error: code = NotFound desc = could not find container \"f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225\": container with ID starting with f1a240ed264a3a1684aff2ec5a459432c208108aef420d5388e3e30f313da225 not found: ID does not exist" Oct 31 00:24:47.510743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.510713 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n"] Oct 31 00:24:47.559772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.559742 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-wnf8n"] Oct 31 00:24:47.735579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:47.735540 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" path="/var/lib/kubelet/pods/194e388d-96b0-4a1c-9124-3daccc692b4f/volumes" Oct 31 00:24:53.319666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:53.319631 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:24:53.472372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:53.472294 4969 generic.go:356] "Generic (PLEG): container finished" podID="58101b4a-2aef-45e9-b534-89351404d96e" containerID="6d86b158f7cfce17d284645ce3cd74e7097ee1c3f86489d0507fb842dbc5215f" exitCode=0 Oct 31 00:24:53.472518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:53.472361 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bnwts-jlqxx" event={"ID":"58101b4a-2aef-45e9-b534-89351404d96e","Type":"ContainerDied","Data":"6d86b158f7cfce17d284645ce3cd74e7097ee1c3f86489d0507fb842dbc5215f"} Oct 31 00:24:54.629835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.629809 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:24:54.719881 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.719851 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58101b4a-2aef-45e9-b534-89351404d96e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58101b4a-2aef-45e9-b534-89351404d96e-archives-path\") pod \"58101b4a-2aef-45e9-b534-89351404d96e\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " Oct 31 00:24:54.720021 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.719975 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-service-ca-bundle\") pod \"58101b4a-2aef-45e9-b534-89351404d96e\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " Oct 31 00:24:54.720077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.720019 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-trusted-ca-bundle\") pod \"58101b4a-2aef-45e9-b534-89351404d96e\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " Oct 31 00:24:54.720138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.720121 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58101b4a-2aef-45e9-b534-89351404d96e-kube-api-access-lvsr8\" (UniqueName: \"kubernetes.io/projected/58101b4a-2aef-45e9-b534-89351404d96e-kube-api-access-lvsr8\") pod \"58101b4a-2aef-45e9-b534-89351404d96e\" (UID: \"58101b4a-2aef-45e9-b534-89351404d96e\") " Oct 31 00:24:54.720249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.720172 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58101b4a-2aef-45e9-b534-89351404d96e-archives-path" pod "58101b4a-2aef-45e9-b534-89351404d96e" (UID: "58101b4a-2aef-45e9-b534-89351404d96e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.720376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.720355 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-trusted-ca-bundle" pod "58101b4a-2aef-45e9-b534-89351404d96e" (UID: "58101b4a-2aef-45e9-b534-89351404d96e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:54.720417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.720352 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-service-ca-bundle" pod "58101b4a-2aef-45e9-b534-89351404d96e" (UID: "58101b4a-2aef-45e9-b534-89351404d96e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:54.720515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.720499 4969 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/58101b4a-2aef-45e9-b534-89351404d96e-archives-path\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.720560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.720520 4969 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-service-ca-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.720560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.720529 4969 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58101b4a-2aef-45e9-b534-89351404d96e-trusted-ca-bundle\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.722449 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.722427 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58101b4a-2aef-45e9-b534-89351404d96e-kube-api-access-lvsr8" pod "58101b4a-2aef-45e9-b534-89351404d96e" (UID: "58101b4a-2aef-45e9-b534-89351404d96e"). InnerVolumeSpecName "kube-api-access-lvsr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:54.821517 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:54.821484 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvsr8\" (UniqueName: \"kubernetes.io/projected/58101b4a-2aef-45e9-b534-89351404d96e-kube-api-access-lvsr8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:55.490244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:55.490213 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bnwts-jlqxx" Oct 31 00:24:55.490244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:55.490233 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bnwts-jlqxx" event={"ID":"58101b4a-2aef-45e9-b534-89351404d96e","Type":"ContainerDied","Data":"722a720b670e712e0d550516e2b1d731d5f4c2d09760e3b876722bd4c037abf2"} Oct 31 00:24:55.490572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:55.490286 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="722a720b670e712e0d550516e2b1d731d5f4c2d09760e3b876722bd4c037abf2" Oct 31 00:24:57.628675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.628638 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" podUID="b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" containerName="main" containerID="cri-o://b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6" gracePeriod=45 Oct 31 00:24:57.748844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.748805 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" podUID="b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": dial tcp 10.133.0.97:8000: connect: connection refused" Oct 31 00:24:57.909979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.909952 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:24:57.955355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.955318 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-kube-api-access-9795h\" (UniqueName: \"kubernetes.io/projected/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-kube-api-access-9795h\") pod \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " Oct 31 00:24:57.955535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.955380 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-home\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-home\") pod \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " Oct 31 00:24:57.955535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.955403 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-dshm\") pod \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " Oct 31 00:24:57.955535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.955478 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-model-cache\") pod \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " Oct 31 00:24:57.955535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.955526 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tmp-dir\") pod \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " Oct 31 00:24:57.955808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.955543 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tls-certs\") pod \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\" (UID: \"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2\") " Oct 31 00:24:57.955808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.955741 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-home" pod "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" (UID: "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:57.955808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.955779 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-model-cache" pod "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" (UID: "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:57.955956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.955843 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tmp-dir" pod "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" (UID: "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:57.956050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.956034 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:57.956105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.956053 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:57.956105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.956081 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:57.957888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.957865 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-kube-api-access-9795h" pod "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" (UID: "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2"). InnerVolumeSpecName "kube-api-access-9795h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:57.957999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.957888 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tls-certs" pod "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" (UID: "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:24:57.957999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:57.957918 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-dshm" pod "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" (UID: "b92eaeda-fb10-41fc-afcc-b7b53b74e2d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:58.057433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.057402 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9795h\" (UniqueName: \"kubernetes.io/projected/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-kube-api-access-9795h\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:58.057433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.057429 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:58.057433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.057439 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:24:58.514944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.514914 4969 generic.go:356] "Generic (PLEG): container finished" podID="b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" containerID="b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6" exitCode=0 Oct 31 00:24:58.515113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.514958 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" Oct 31 00:24:58.515113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.514993 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" event={"ID":"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2","Type":"ContainerDied","Data":"b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6"} Oct 31 00:24:58.515113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.515043 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt" event={"ID":"b92eaeda-fb10-41fc-afcc-b7b53b74e2d2","Type":"ContainerDied","Data":"6833f132409b7d5fb9a790413416dbbec446b97f0c59e22c919d6326b19827c4"} Oct 31 00:24:58.515113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.515065 4969 scope.go:122] "RemoveContainer" containerID="b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6" Oct 31 00:24:58.526375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.526359 4969 scope.go:122] "RemoveContainer" containerID="b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6" Oct 31 00:24:58.526677 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:24:58.526657 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6\": container with ID starting with b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6 not found: ID does not exist" containerID="b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6" Oct 31 00:24:58.526756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.526689 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6"} err="failed to get container status \"b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6\": rpc error: code = NotFound desc = could not find container \"b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6\": container with ID starting with b41ab19eeb1d1e51b6cdb76ea8285a819902287e0f168d901bd43be7d58fabc6 not found: ID does not exist" Oct 31 00:24:58.680911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.680867 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt"] Oct 31 00:24:58.730362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:58.730333 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-b-d925972c-kserve-dddd97b7d-sb7dt"] Oct 31 00:24:59.734099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:24:59.734057 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" path="/var/lib/kubelet/pods/b92eaeda-fb10-41fc-afcc-b7b53b74e2d2/volumes" Oct 31 00:25:02.727502 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:25:02.727464 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:25:05.184537 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.184496 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb"] Oct 31 00:25:05.185269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.185247 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" containerName="main" Oct 31 00:25:05.185376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.185271 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" containerName="main" Oct 31 00:25:05.185376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.185303 4969 cpu_manager.go:414] "containerMap: removing container" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" Oct 31 00:25:05.185376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.185311 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" Oct 31 00:25:05.185376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.185323 4969 cpu_manager.go:414] "containerMap: removing container" podUID="58101b4a-2aef-45e9-b534-89351404d96e" containerName="insights-gathering" Oct 31 00:25:05.185376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.185331 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58101b4a-2aef-45e9-b534-89351404d96e" containerName="insights-gathering" Oct 31 00:25:05.185654 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.185490 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="194e388d-96b0-4a1c-9124-3daccc692b4f" containerName="payload-processing" Oct 31 00:25:05.185654 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.185503 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="58101b4a-2aef-45e9-b534-89351404d96e" containerName="insights-gathering" Oct 31 00:25:05.185654 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.185522 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92eaeda-fb10-41fc-afcc-b7b53b74e2d2" containerName="main" Oct 31 00:25:05.190407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.190385 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.247037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.247007 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb"] Oct 31 00:25:05.291235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.291207 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:25:05.291235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.291233 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:25:05.339151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.339122 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8899f9f-mg2gb\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.339341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.339314 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8qn2\" (UniqueName: \"kubernetes.io/projected/39467d37-07ab-4e4f-a519-5ea4b08ced73-kube-api-access-f8qn2\") pod \"maas-api-revoke-sub-e8899f9f-mg2gb\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.339434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.339420 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8899f9f-mg2gb\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.440503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.440425 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8899f9f-mg2gb\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.440679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.440554 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8qn2\" (UniqueName: \"kubernetes.io/projected/39467d37-07ab-4e4f-a519-5ea4b08ced73-kube-api-access-f8qn2\") pod \"maas-api-revoke-sub-e8899f9f-mg2gb\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.440679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.440633 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8899f9f-mg2gb\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.441281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.441261 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8899f9f-mg2gb\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.442953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.442936 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8899f9f-mg2gb\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.501559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.501527 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8qn2\" (UniqueName: \"kubernetes.io/projected/39467d37-07ab-4e4f-a519-5ea4b08ced73-kube-api-access-f8qn2\") pod \"maas-api-revoke-sub-e8899f9f-mg2gb\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.504358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.504339 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:05.885352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:05.885321 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb"] Oct 31 00:25:05.886057 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:25:05.886028 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39467d37_07ab_4e4f_a519_5ea4b08ced73.slice/crio-fc994d4e1df050bd072e757882f0149235d90561379e65b0cb70a83edb02fdd1 WatchSource:0}: Error finding container fc994d4e1df050bd072e757882f0149235d90561379e65b0cb70a83edb02fdd1: Status 404 returned error can't find the container with id fc994d4e1df050bd072e757882f0149235d90561379e65b0cb70a83edb02fdd1 Oct 31 00:25:06.573785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:06.573696 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" event={"ID":"39467d37-07ab-4e4f-a519-5ea4b08ced73","Type":"ContainerStarted","Data":"62e2cde4ca5870e8ba42327c6449aa8769bacae0af80ecc42280d140118e2e75"} Oct 31 00:25:06.573785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:06.573748 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" event={"ID":"39467d37-07ab-4e4f-a519-5ea4b08ced73","Type":"ContainerStarted","Data":"fc994d4e1df050bd072e757882f0149235d90561379e65b0cb70a83edb02fdd1"} Oct 31 00:25:06.691096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:06.691044 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" podStartSLOduration=1.691028832 podStartE2EDuration="1.691028832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:06.688493808 +0000 UTC m=+1339.163852231" watchObservedRunningTime="2026-10-31 00:25:06.691028832 +0000 UTC m=+1339.166387254" Oct 31 00:25:07.581551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:07.581515 4969 generic.go:356] "Generic (PLEG): container finished" podID="39467d37-07ab-4e4f-a519-5ea4b08ced73" containerID="62e2cde4ca5870e8ba42327c6449aa8769bacae0af80ecc42280d140118e2e75" exitCode=0 Oct 31 00:25:07.581973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:07.581627 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" event={"ID":"39467d37-07ab-4e4f-a519-5ea4b08ced73","Type":"ContainerDied","Data":"62e2cde4ca5870e8ba42327c6449aa8769bacae0af80ecc42280d140118e2e75"} Oct 31 00:25:08.741034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.741010 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:08.878480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.878444 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39467d37-07ab-4e4f-a519-5ea4b08ced73-kube-api-access-f8qn2\" (UniqueName: \"kubernetes.io/projected/39467d37-07ab-4e4f-a519-5ea4b08ced73-kube-api-access-f8qn2\") pod \"39467d37-07ab-4e4f-a519-5ea4b08ced73\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " Oct 31 00:25:08.878800 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.878770 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-openshift-ca\") pod \"39467d37-07ab-4e4f-a519-5ea4b08ced73\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " Oct 31 00:25:08.878932 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.878838 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-xks-ca\") pod \"39467d37-07ab-4e4f-a519-5ea4b08ced73\" (UID: \"39467d37-07ab-4e4f-a519-5ea4b08ced73\") " Oct 31 00:25:08.879530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.879139 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-openshift-ca" pod "39467d37-07ab-4e4f-a519-5ea4b08ced73" (UID: "39467d37-07ab-4e4f-a519-5ea4b08ced73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:08.879530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.879259 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:08.881288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.881267 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39467d37-07ab-4e4f-a519-5ea4b08ced73-kube-api-access-f8qn2" pod "39467d37-07ab-4e4f-a519-5ea4b08ced73" (UID: "39467d37-07ab-4e4f-a519-5ea4b08ced73"). InnerVolumeSpecName "kube-api-access-f8qn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:08.881542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.881526 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-xks-ca" pod "39467d37-07ab-4e4f-a519-5ea4b08ced73" (UID: "39467d37-07ab-4e4f-a519-5ea4b08ced73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:08.980262 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.980230 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39467d37-07ab-4e4f-a519-5ea4b08ced73-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:08.980262 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:08.980258 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8qn2\" (UniqueName: \"kubernetes.io/projected/39467d37-07ab-4e4f-a519-5ea4b08ced73-kube-api-access-f8qn2\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:09.597845 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:09.597811 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" Oct 31 00:25:09.597845 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:09.597834 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb" event={"ID":"39467d37-07ab-4e4f-a519-5ea4b08ced73","Type":"ContainerDied","Data":"fc994d4e1df050bd072e757882f0149235d90561379e65b0cb70a83edb02fdd1"} Oct 31 00:25:09.598072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:09.597912 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc994d4e1df050bd072e757882f0149235d90561379e65b0cb70a83edb02fdd1" Oct 31 00:25:16.012246 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.012196 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k"] Oct 31 00:25:16.013779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.013753 4969 cpu_manager.go:414] "containerMap: removing container" podUID="39467d37-07ab-4e4f-a519-5ea4b08ced73" containerName="revoke-keys" Oct 31 00:25:16.013936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.013920 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39467d37-07ab-4e4f-a519-5ea4b08ced73" containerName="revoke-keys" Oct 31 00:25:16.014181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.014165 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="39467d37-07ab-4e4f-a519-5ea4b08ced73" containerName="revoke-keys" Oct 31 00:25:16.018899 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.018874 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.071894 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.071799 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k"] Oct 31 00:25:16.115808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.115750 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:25:16.115974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.115886 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:25:16.177581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.177543 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45ad2e0-bng2k\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.177776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.177702 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45ad2e0-bng2k\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.177865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.177848 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtxcm\" (UniqueName: \"kubernetes.io/projected/3add4aa1-b87d-426a-9dce-c24a28f0b608-kube-api-access-qtxcm\") pod \"maas-api-revoke-sub-a45ad2e0-bng2k\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.279170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.279079 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtxcm\" (UniqueName: \"kubernetes.io/projected/3add4aa1-b87d-426a-9dce-c24a28f0b608-kube-api-access-qtxcm\") pod \"maas-api-revoke-sub-a45ad2e0-bng2k\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.279170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.279166 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45ad2e0-bng2k\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.279396 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.279208 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45ad2e0-bng2k\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.279812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.279794 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45ad2e0-bng2k\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.281671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.281652 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45ad2e0-bng2k\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.340661 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.340626 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtxcm\" (UniqueName: \"kubernetes.io/projected/3add4aa1-b87d-426a-9dce-c24a28f0b608-kube-api-access-qtxcm\") pod \"maas-api-revoke-sub-a45ad2e0-bng2k\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:16.638264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:16.638230 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:17.020719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:17.020685 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k"] Oct 31 00:25:17.021249 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:25:17.021225 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3add4aa1_b87d_426a_9dce_c24a28f0b608.slice/crio-4f3a51cc77ff59f8b32caf5e8d52851ff327168dd311373dfc84d87ec623b7a0 WatchSource:0}: Error finding container 4f3a51cc77ff59f8b32caf5e8d52851ff327168dd311373dfc84d87ec623b7a0: Status 404 returned error can't find the container with id 4f3a51cc77ff59f8b32caf5e8d52851ff327168dd311373dfc84d87ec623b7a0 Oct 31 00:25:17.301750 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:17.301679 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68"] Oct 31 00:25:17.680018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:17.679978 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" event={"ID":"3add4aa1-b87d-426a-9dce-c24a28f0b608","Type":"ContainerStarted","Data":"8277dec9d1c7906cf0eec334a5817e154105d8e5833a5eeee645e67482c55c97"} Oct 31 00:25:17.680190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:17.680065 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" event={"ID":"3add4aa1-b87d-426a-9dce-c24a28f0b608","Type":"ContainerStarted","Data":"4f3a51cc77ff59f8b32caf5e8d52851ff327168dd311373dfc84d87ec623b7a0"} Oct 31 00:25:17.796194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:17.796147 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" podStartSLOduration=2.796129987 podStartE2EDuration="2.796129987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:17.795492967 +0000 UTC m=+1350.270851392" watchObservedRunningTime="2026-10-31 00:25:17.796129987 +0000 UTC m=+1350.271488412" Oct 31 00:25:18.688320 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:18.688285 4969 generic.go:356] "Generic (PLEG): container finished" podID="3add4aa1-b87d-426a-9dce-c24a28f0b608" containerID="8277dec9d1c7906cf0eec334a5817e154105d8e5833a5eeee645e67482c55c97" exitCode=0 Oct 31 00:25:18.688793 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:18.688376 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" event={"ID":"3add4aa1-b87d-426a-9dce-c24a28f0b608","Type":"ContainerDied","Data":"8277dec9d1c7906cf0eec334a5817e154105d8e5833a5eeee645e67482c55c97"} Oct 31 00:25:19.870453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:19.870429 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:19.922890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:19.922862 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-xks-ca\") pod \"3add4aa1-b87d-426a-9dce-c24a28f0b608\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " Oct 31 00:25:19.923022 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:19.922985 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3add4aa1-b87d-426a-9dce-c24a28f0b608-kube-api-access-qtxcm\" (UniqueName: \"kubernetes.io/projected/3add4aa1-b87d-426a-9dce-c24a28f0b608-kube-api-access-qtxcm\") pod \"3add4aa1-b87d-426a-9dce-c24a28f0b608\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " Oct 31 00:25:19.923124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:19.923109 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-openshift-ca\") pod \"3add4aa1-b87d-426a-9dce-c24a28f0b608\" (UID: \"3add4aa1-b87d-426a-9dce-c24a28f0b608\") " Oct 31 00:25:19.923499 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:19.923471 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-openshift-ca" pod "3add4aa1-b87d-426a-9dce-c24a28f0b608" (UID: "3add4aa1-b87d-426a-9dce-c24a28f0b608"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:19.925162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:19.925131 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-xks-ca" pod "3add4aa1-b87d-426a-9dce-c24a28f0b608" (UID: "3add4aa1-b87d-426a-9dce-c24a28f0b608"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:19.925257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:19.925238 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3add4aa1-b87d-426a-9dce-c24a28f0b608-kube-api-access-qtxcm" pod "3add4aa1-b87d-426a-9dce-c24a28f0b608" (UID: "3add4aa1-b87d-426a-9dce-c24a28f0b608"). InnerVolumeSpecName "kube-api-access-qtxcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:20.025104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:20.025036 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtxcm\" (UniqueName: \"kubernetes.io/projected/3add4aa1-b87d-426a-9dce-c24a28f0b608-kube-api-access-qtxcm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:20.025104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:20.025062 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:20.025104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:20.025072 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3add4aa1-b87d-426a-9dce-c24a28f0b608-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:20.704923 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:20.704880 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" event={"ID":"3add4aa1-b87d-426a-9dce-c24a28f0b608","Type":"ContainerDied","Data":"4f3a51cc77ff59f8b32caf5e8d52851ff327168dd311373dfc84d87ec623b7a0"} Oct 31 00:25:20.705094 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:20.704944 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f3a51cc77ff59f8b32caf5e8d52851ff327168dd311373dfc84d87ec623b7a0" Oct 31 00:25:20.705094 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:20.704898 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k" Oct 31 00:25:22.868794 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.868764 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt"] Oct 31 00:25:22.869328 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.869313 4969 cpu_manager.go:414] "containerMap: removing container" podUID="3add4aa1-b87d-426a-9dce-c24a28f0b608" containerName="revoke-keys" Oct 31 00:25:22.869391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.869329 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3add4aa1-b87d-426a-9dce-c24a28f0b608" containerName="revoke-keys" Oct 31 00:25:22.869437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.869428 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="3add4aa1-b87d-426a-9dce-c24a28f0b608" containerName="revoke-keys" Oct 31 00:25:22.873013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.872995 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.959836 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.959799 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.960017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.959847 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.960017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.959902 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.960017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.959927 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da24b1-7783-4b4e-acb0-df82e0165b87-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.960017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.959966 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.960017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.960005 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.960435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.960045 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.960435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.960209 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.960435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.960240 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6l6j\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-kube-api-access-d6l6j\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:22.960688 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:22.960663 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt"] Oct 31 00:25:23.060955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.060925 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.060963 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6l6j\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-kube-api-access-d6l6j\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061000 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061026 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061079 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061198 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da24b1-7783-4b4e-acb0-df82e0165b87-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061277 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061326 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061366 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061450 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061486 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061365 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061644 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.061894 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.061874 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da24b1-7783-4b4e-acb0-df82e0165b87-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.063739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.063718 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.063863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.063746 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.122121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.122043 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6l6j\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-kube-api-access-d6l6j\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.122121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.122060 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.187153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.187124 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:23.620332 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:25:23.620302 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79da24b1_7783_4b4e_acb0_df82e0165b87.slice/crio-07da8583b436d371b9ec67866bf7843118e2f48f19ff8a547d808744b58842e2 WatchSource:0}: Error finding container 07da8583b436d371b9ec67866bf7843118e2f48f19ff8a547d808744b58842e2: Status 404 returned error can't find the container with id 07da8583b436d371b9ec67866bf7843118e2f48f19ff8a547d808744b58842e2 Oct 31 00:25:23.620439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.620333 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt"] Oct 31 00:25:23.622651 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.622621 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:25:23.622741 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.622701 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:25:23.622741 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.622733 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:25:23.734977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.734945 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" event={"ID":"79da24b1-7783-4b4e-acb0-df82e0165b87","Type":"ContainerStarted","Data":"12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649"} Oct 31 00:25:23.735131 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.735000 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" event={"ID":"79da24b1-7783-4b4e-acb0-df82e0165b87","Type":"ContainerStarted","Data":"07da8583b436d371b9ec67866bf7843118e2f48f19ff8a547d808744b58842e2"} Oct 31 00:25:23.853261 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:23.853198 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" podStartSLOduration=1.853181655 podStartE2EDuration="1.853181655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:23.850089879 +0000 UTC m=+1356.325448303" watchObservedRunningTime="2026-10-31 00:25:23.853181655 +0000 UTC m=+1356.328540081" Oct 31 00:25:24.188292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:24.188205 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:24.191277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:24.190915 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:24.191277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:24.190977 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:25.197009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:25.196973 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:25.197745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:25.197710 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:26.197620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:26.193024 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:26.197620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:26.193090 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:27.189992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:27.189953 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:27.190274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:27.190241 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:28.197633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.197573 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:28.198214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.198184 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:28.504074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.503993 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr"] Oct 31 00:25:28.508341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.508316 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.553632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.553584 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr"] Oct 31 00:25:28.561917 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.561887 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:25:28.601755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.600886 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:25:28.649167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.649137 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b475ef40-fstxr\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.649336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.649300 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b475ef40-fstxr\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.649379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.649350 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnv88\" (UniqueName: \"kubernetes.io/projected/4172a010-d35c-43b6-9234-b867c7227f25-kube-api-access-xnv88\") pod \"maas-api-revoke-sub-b475ef40-fstxr\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.750847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.750814 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b475ef40-fstxr\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.750847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.750853 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnv88\" (UniqueName: \"kubernetes.io/projected/4172a010-d35c-43b6-9234-b867c7227f25-kube-api-access-xnv88\") pod \"maas-api-revoke-sub-b475ef40-fstxr\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.751075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.751011 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b475ef40-fstxr\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.751485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.751461 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b475ef40-fstxr\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.753604 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.753571 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b475ef40-fstxr\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.813046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.812968 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnv88\" (UniqueName: \"kubernetes.io/projected/4172a010-d35c-43b6-9234-b867c7227f25-kube-api-access-xnv88\") pod \"maas-api-revoke-sub-b475ef40-fstxr\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:28.822935 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:28.822906 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:29.201620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:29.194998 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:29.201620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:29.195057 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:29.230729 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:25:29.230695 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4172a010_d35c_43b6_9234_b867c7227f25.slice/crio-ccfcccbd488e9a52c2296423691302a6606e826143132b29cf311f0dfec3f77d WatchSource:0}: Error finding container ccfcccbd488e9a52c2296423691302a6606e826143132b29cf311f0dfec3f77d: Status 404 returned error can't find the container with id ccfcccbd488e9a52c2296423691302a6606e826143132b29cf311f0dfec3f77d Oct 31 00:25:29.232047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:29.232021 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr"] Oct 31 00:25:29.777097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:29.777042 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" event={"ID":"4172a010-d35c-43b6-9234-b867c7227f25","Type":"ContainerStarted","Data":"e3dd35228f5e56597bf0aa7fe03c570e3a3d3ce8cf7fad4d59227143b8f9f4fd"} Oct 31 00:25:29.777271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:29.777114 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" event={"ID":"4172a010-d35c-43b6-9234-b867c7227f25","Type":"ContainerStarted","Data":"ccfcccbd488e9a52c2296423691302a6606e826143132b29cf311f0dfec3f77d"} Oct 31 00:25:29.900240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:29.900185 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" podStartSLOduration=1.900170856 podStartE2EDuration="1.900170856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:29.897945177 +0000 UTC m=+1362.373303600" watchObservedRunningTime="2026-10-31 00:25:29.900170856 +0000 UTC m=+1362.375529280" Oct 31 00:25:30.187898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:30.187867 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:30.188046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:30.187926 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:30.786330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:30.786281 4969 generic.go:356] "Generic (PLEG): container finished" podID="4172a010-d35c-43b6-9234-b867c7227f25" containerID="e3dd35228f5e56597bf0aa7fe03c570e3a3d3ce8cf7fad4d59227143b8f9f4fd" exitCode=0 Oct 31 00:25:30.786763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:30.786344 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" event={"ID":"4172a010-d35c-43b6-9234-b867c7227f25","Type":"ContainerDied","Data":"e3dd35228f5e56597bf0aa7fe03c570e3a3d3ce8cf7fad4d59227143b8f9f4fd"} Oct 31 00:25:31.192242 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:31.192214 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:31.192728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:31.192708 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:31.193436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:31.193417 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:25:31.944784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:31.944761 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:32.058088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.058018 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b"] Oct 31 00:25:32.061648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.061623 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fa72829-cr77b"] Oct 31 00:25:32.093236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.093214 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-xks-ca\") pod \"4172a010-d35c-43b6-9234-b867c7227f25\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " Oct 31 00:25:32.093360 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.093308 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-openshift-ca\") pod \"4172a010-d35c-43b6-9234-b867c7227f25\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " Oct 31 00:25:32.093360 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.093354 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4172a010-d35c-43b6-9234-b867c7227f25-kube-api-access-xnv88\" (UniqueName: \"kubernetes.io/projected/4172a010-d35c-43b6-9234-b867c7227f25-kube-api-access-xnv88\") pod \"4172a010-d35c-43b6-9234-b867c7227f25\" (UID: \"4172a010-d35c-43b6-9234-b867c7227f25\") " Oct 31 00:25:32.093730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.093708 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-openshift-ca" pod "4172a010-d35c-43b6-9234-b867c7227f25" (UID: "4172a010-d35c-43b6-9234-b867c7227f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:32.093879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.093864 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:32.095742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.095718 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-xks-ca" pod "4172a010-d35c-43b6-9234-b867c7227f25" (UID: "4172a010-d35c-43b6-9234-b867c7227f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:32.095826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.095797 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4172a010-d35c-43b6-9234-b867c7227f25-kube-api-access-xnv88" pod "4172a010-d35c-43b6-9234-b867c7227f25" (UID: "4172a010-d35c-43b6-9234-b867c7227f25"). InnerVolumeSpecName "kube-api-access-xnv88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:32.195318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.195282 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4172a010-d35c-43b6-9234-b867c7227f25-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:32.195318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.195310 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnv88\" (UniqueName: \"kubernetes.io/projected/4172a010-d35c-43b6-9234-b867c7227f25-kube-api-access-xnv88\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:32.322335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.322232 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" podUID="0ca34f89-54dd-401d-8007-d4ffe2951b73" containerName="main" containerID="cri-o://fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91" gracePeriod=45 Oct 31 00:25:32.394909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.394880 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b"] Oct 31 00:25:32.395475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.395458 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4172a010-d35c-43b6-9234-b867c7227f25" containerName="revoke-keys" Oct 31 00:25:32.395576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.395478 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4172a010-d35c-43b6-9234-b867c7227f25" containerName="revoke-keys" Oct 31 00:25:32.395681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.395671 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4172a010-d35c-43b6-9234-b867c7227f25" containerName="revoke-keys" Oct 31 00:25:32.399156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.399138 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.457173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.457140 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b"] Oct 31 00:25:32.499148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.499113 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5e2cf70-x4l7b\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.499574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.499291 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4m9\" (UniqueName: \"kubernetes.io/projected/93f6f609-2edb-40cc-bdf9-2d814ff68891-kube-api-access-wb4m9\") pod \"maas-api-revoke-sub-e5e2cf70-x4l7b\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.499574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.499423 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5e2cf70-x4l7b\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.586527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.586504 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:25:32.600660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.600636 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5e2cf70-x4l7b\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.600768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.600755 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4m9\" (UniqueName: \"kubernetes.io/projected/93f6f609-2edb-40cc-bdf9-2d814ff68891-kube-api-access-wb4m9\") pod \"maas-api-revoke-sub-e5e2cf70-x4l7b\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.600838 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.600823 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5e2cf70-x4l7b\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.601361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.601340 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5e2cf70-x4l7b\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.603111 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.603093 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5e2cf70-x4l7b\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.662080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.662049 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4m9\" (UniqueName: \"kubernetes.io/projected/93f6f609-2edb-40cc-bdf9-2d814ff68891-kube-api-access-wb4m9\") pod \"maas-api-revoke-sub-e5e2cf70-x4l7b\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.701294 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.701249 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca34f89-54dd-401d-8007-d4ffe2951b73-kube-api-access-nrb5z\" (UniqueName: \"kubernetes.io/projected/0ca34f89-54dd-401d-8007-d4ffe2951b73-kube-api-access-nrb5z\") pod \"0ca34f89-54dd-401d-8007-d4ffe2951b73\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " Oct 31 00:25:32.701452 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.701331 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-model-cache\") pod \"0ca34f89-54dd-401d-8007-d4ffe2951b73\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " Oct 31 00:25:32.701452 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.701445 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca34f89-54dd-401d-8007-d4ffe2951b73-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca34f89-54dd-401d-8007-d4ffe2951b73-tls-certs\") pod \"0ca34f89-54dd-401d-8007-d4ffe2951b73\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " Oct 31 00:25:32.701543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.701484 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-home\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-home\") pod \"0ca34f89-54dd-401d-8007-d4ffe2951b73\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " Oct 31 00:25:32.701617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.701541 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-tmp-dir\") pod \"0ca34f89-54dd-401d-8007-d4ffe2951b73\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " Oct 31 00:25:32.701679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.701580 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-model-cache" pod "0ca34f89-54dd-401d-8007-d4ffe2951b73" (UID: "0ca34f89-54dd-401d-8007-d4ffe2951b73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:32.701734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.701724 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-dshm\") pod \"0ca34f89-54dd-401d-8007-d4ffe2951b73\" (UID: \"0ca34f89-54dd-401d-8007-d4ffe2951b73\") " Oct 31 00:25:32.702223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.702192 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:32.704030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.703994 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca34f89-54dd-401d-8007-d4ffe2951b73-kube-api-access-nrb5z" pod "0ca34f89-54dd-401d-8007-d4ffe2951b73" (UID: "0ca34f89-54dd-401d-8007-d4ffe2951b73"). InnerVolumeSpecName "kube-api-access-nrb5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:32.705730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.705688 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-tmp-dir" pod "0ca34f89-54dd-401d-8007-d4ffe2951b73" (UID: "0ca34f89-54dd-401d-8007-d4ffe2951b73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:32.705730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.705701 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-home" pod "0ca34f89-54dd-401d-8007-d4ffe2951b73" (UID: "0ca34f89-54dd-401d-8007-d4ffe2951b73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:32.707940 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.707890 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca34f89-54dd-401d-8007-d4ffe2951b73-tls-certs" pod "0ca34f89-54dd-401d-8007-d4ffe2951b73" (UID: "0ca34f89-54dd-401d-8007-d4ffe2951b73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:32.708239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.708218 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-dshm" pod "0ca34f89-54dd-401d-8007-d4ffe2951b73" (UID: "0ca34f89-54dd-401d-8007-d4ffe2951b73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:32.713562 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.713540 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:32.735554 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:25:32.735523 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:25:32.806160 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.806127 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:32.806332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.806167 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrb5z\" (UniqueName: \"kubernetes.io/projected/0ca34f89-54dd-401d-8007-d4ffe2951b73-kube-api-access-nrb5z\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:32.806332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.806182 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca34f89-54dd-401d-8007-d4ffe2951b73-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:32.806332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.806195 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:32.806332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.806205 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca34f89-54dd-401d-8007-d4ffe2951b73-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:32.812645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.812583 4969 generic.go:356] "Generic (PLEG): container finished" podID="0ca34f89-54dd-401d-8007-d4ffe2951b73" containerID="fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91" exitCode=0 Oct 31 00:25:32.812780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.812724 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" Oct 31 00:25:32.812780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.812771 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" event={"ID":"0ca34f89-54dd-401d-8007-d4ffe2951b73","Type":"ContainerDied","Data":"fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91"} Oct 31 00:25:32.812897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.812848 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68" event={"ID":"0ca34f89-54dd-401d-8007-d4ffe2951b73","Type":"ContainerDied","Data":"0c38c25dc22297809933f5f34844f9f35a495bcee7cd7dd28b65d0f77675bd25"} Oct 31 00:25:32.812897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.812878 4969 scope.go:122] "RemoveContainer" containerID="fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91" Oct 31 00:25:32.818018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.815394 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" Oct 31 00:25:32.818018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.815430 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr" event={"ID":"4172a010-d35c-43b6-9234-b867c7227f25","Type":"ContainerDied","Data":"ccfcccbd488e9a52c2296423691302a6606e826143132b29cf311f0dfec3f77d"} Oct 31 00:25:32.818018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.815516 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccfcccbd488e9a52c2296423691302a6606e826143132b29cf311f0dfec3f77d" Oct 31 00:25:32.827020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.827004 4969 scope.go:122] "RemoveContainer" containerID="fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91" Oct 31 00:25:32.827303 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:25:32.827281 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91\": container with ID starting with fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91 not found: ID does not exist" containerID="fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91" Oct 31 00:25:32.827383 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:32.827316 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91"} err="failed to get container status \"fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91\": rpc error: code = NotFound desc = could not find container \"fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91\": container with ID starting with fa6529fcf2f1b4c725c07ed45fee35b18613a92f775b1873bf87b8ae6aa2ee91 not found: ID does not exist" Oct 31 00:25:33.030947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:33.030911 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68"] Oct 31 00:25:33.037904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:33.037869 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/e2e-conflict-a-d925972c-kserve-5688fdf49b-tdt68"] Oct 31 00:25:33.095534 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:33.095505 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b"] Oct 31 00:25:33.096139 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:25:33.096109 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f6f609_2edb_40cc_bdf9_2d814ff68891.slice/crio-1464daa571cef1ed105b4b8988d2f42ea5fff83f0e277833aec5016df10f2feb WatchSource:0}: Error finding container 1464daa571cef1ed105b4b8988d2f42ea5fff83f0e277833aec5016df10f2feb: Status 404 returned error can't find the container with id 1464daa571cef1ed105b4b8988d2f42ea5fff83f0e277833aec5016df10f2feb Oct 31 00:25:33.735323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:33.735286 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00829b05-9424-4102-bfcc-2a726893b7f3" path="/var/lib/kubelet/pods/00829b05-9424-4102-bfcc-2a726893b7f3/volumes" Oct 31 00:25:33.735812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:33.735794 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca34f89-54dd-401d-8007-d4ffe2951b73" path="/var/lib/kubelet/pods/0ca34f89-54dd-401d-8007-d4ffe2951b73/volumes" Oct 31 00:25:33.824682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:33.824643 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" event={"ID":"93f6f609-2edb-40cc-bdf9-2d814ff68891","Type":"ContainerStarted","Data":"c8b69aa06a5d79dc9c1a04120e0c98206972cd239b8dfce660639f0c65d30f7d"} Oct 31 00:25:33.824863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:33.824699 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" event={"ID":"93f6f609-2edb-40cc-bdf9-2d814ff68891","Type":"ContainerStarted","Data":"1464daa571cef1ed105b4b8988d2f42ea5fff83f0e277833aec5016df10f2feb"} Oct 31 00:25:33.943662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:33.943610 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" podStartSLOduration=1.943579212 podStartE2EDuration="1.943579212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:33.939061651 +0000 UTC m=+1366.414420111" watchObservedRunningTime="2026-10-31 00:25:33.943579212 +0000 UTC m=+1366.418937629" Oct 31 00:25:34.840526 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:34.840491 4969 generic.go:356] "Generic (PLEG): container finished" podID="93f6f609-2edb-40cc-bdf9-2d814ff68891" containerID="c8b69aa06a5d79dc9c1a04120e0c98206972cd239b8dfce660639f0c65d30f7d" exitCode=0 Oct 31 00:25:34.841162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:34.840875 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" event={"ID":"93f6f609-2edb-40cc-bdf9-2d814ff68891","Type":"ContainerDied","Data":"c8b69aa06a5d79dc9c1a04120e0c98206972cd239b8dfce660639f0c65d30f7d"} Oct 31 00:25:36.223313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.223272 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:36.382623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.380504 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-xks-ca\") pod \"93f6f609-2edb-40cc-bdf9-2d814ff68891\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " Oct 31 00:25:36.382623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.380659 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-openshift-ca\") pod \"93f6f609-2edb-40cc-bdf9-2d814ff68891\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " Oct 31 00:25:36.382623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.380714 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f6f609-2edb-40cc-bdf9-2d814ff68891-kube-api-access-wb4m9\" (UniqueName: \"kubernetes.io/projected/93f6f609-2edb-40cc-bdf9-2d814ff68891-kube-api-access-wb4m9\") pod \"93f6f609-2edb-40cc-bdf9-2d814ff68891\" (UID: \"93f6f609-2edb-40cc-bdf9-2d814ff68891\") " Oct 31 00:25:36.384802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.384773 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-openshift-ca" pod "93f6f609-2edb-40cc-bdf9-2d814ff68891" (UID: "93f6f609-2edb-40cc-bdf9-2d814ff68891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:36.420281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.420225 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-xks-ca" pod "93f6f609-2edb-40cc-bdf9-2d814ff68891" (UID: "93f6f609-2edb-40cc-bdf9-2d814ff68891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:36.421014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.420803 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f6f609-2edb-40cc-bdf9-2d814ff68891-kube-api-access-wb4m9" pod "93f6f609-2edb-40cc-bdf9-2d814ff68891" (UID: "93f6f609-2edb-40cc-bdf9-2d814ff68891"). InnerVolumeSpecName "kube-api-access-wb4m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:36.482702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.482624 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:36.482702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.482653 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4m9\" (UniqueName: \"kubernetes.io/projected/93f6f609-2edb-40cc-bdf9-2d814ff68891-kube-api-access-wb4m9\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:36.482702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.482664 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f6f609-2edb-40cc-bdf9-2d814ff68891-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:36.870079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.870042 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" Oct 31 00:25:36.870079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.870044 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b" event={"ID":"93f6f609-2edb-40cc-bdf9-2d814ff68891","Type":"ContainerDied","Data":"1464daa571cef1ed105b4b8988d2f42ea5fff83f0e277833aec5016df10f2feb"} Oct 31 00:25:36.870312 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:36.870118 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1464daa571cef1ed105b4b8988d2f42ea5fff83f0e277833aec5016df10f2feb" Oct 31 00:25:39.335770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.335738 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj"] Oct 31 00:25:39.336376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.336361 4969 cpu_manager.go:414] "containerMap: removing container" podUID="0ca34f89-54dd-401d-8007-d4ffe2951b73" containerName="main" Oct 31 00:25:39.336417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.336377 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca34f89-54dd-401d-8007-d4ffe2951b73" containerName="main" Oct 31 00:25:39.336417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.336387 4969 cpu_manager.go:414] "containerMap: removing container" podUID="93f6f609-2edb-40cc-bdf9-2d814ff68891" containerName="revoke-keys" Oct 31 00:25:39.336417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.336392 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f6f609-2edb-40cc-bdf9-2d814ff68891" containerName="revoke-keys" Oct 31 00:25:39.336512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.336486 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca34f89-54dd-401d-8007-d4ffe2951b73" containerName="main" Oct 31 00:25:39.336512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.336493 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f6f609-2edb-40cc-bdf9-2d814ff68891" containerName="revoke-keys" Oct 31 00:25:39.340170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.340153 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.442386 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.442356 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0144bd\"/\"distinct-2-w4-0144bd-kserve-self-signed-certs\"" Oct 31 00:25:39.444645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.444619 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj"] Oct 31 00:25:39.517174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.517140 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tmp-dir\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.517365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.517185 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tls-certs\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.517365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.517278 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhrg\" (UniqueName: \"kubernetes.io/projected/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-kube-api-access-6bhrg\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.517484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.517422 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-home\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.517532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.517518 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-model-cache\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.517574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.517557 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-dshm\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.618566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.618537 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhrg\" (UniqueName: \"kubernetes.io/projected/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-kube-api-access-6bhrg\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.618756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.618629 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-home\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.618756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.618666 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-model-cache\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.618756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.618695 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-dshm\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.618756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.618744 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tmp-dir\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.618928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.618768 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tls-certs\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.619078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.619055 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-model-cache\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.619148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.619074 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tmp-dir\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.619223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.619207 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-home\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.621327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.621298 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-dshm\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.621482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.621465 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tls-certs\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.679721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.679697 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhrg\" (UniqueName: \"kubernetes.io/projected/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-kube-api-access-6bhrg\") pod \"distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:39.952877 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:39.952796 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:40.333639 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:25:40.333581 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00e200c4_beab_4ff6_a5cb_c2864b4dfe31.slice/crio-5d9166610dfdca0e34b20498f43293c6b3922e5251e378ae3b39f7c88e484a1a WatchSource:0}: Error finding container 5d9166610dfdca0e34b20498f43293c6b3922e5251e378ae3b39f7c88e484a1a: Status 404 returned error can't find the container with id 5d9166610dfdca0e34b20498f43293c6b3922e5251e378ae3b39f7c88e484a1a Oct 31 00:25:40.333751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:40.333711 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj"] Oct 31 00:25:40.904178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:40.904136 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" event={"ID":"00e200c4-beab-4ff6-a5cb-c2864b4dfe31","Type":"ContainerStarted","Data":"e4c5bfd4da6ef2501b622bf95949fb9dd30d3df15c9c9744d9ab3db493ccecb3"} Oct 31 00:25:40.904178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:40.904177 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" event={"ID":"00e200c4-beab-4ff6-a5cb-c2864b4dfe31","Type":"ContainerStarted","Data":"5d9166610dfdca0e34b20498f43293c6b3922e5251e378ae3b39f7c88e484a1a"} Oct 31 00:25:40.953858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:40.953831 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:41.022780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:41.022728 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podStartSLOduration=2.022713874 podStartE2EDuration="2.022713874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:41.020574754 +0000 UTC m=+1373.495933178" watchObservedRunningTime="2026-10-31 00:25:41.022713874 +0000 UTC m=+1373.498072341" Oct 31 00:25:43.858799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:43.858764 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc"] Oct 31 00:25:43.863025 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:43.863004 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:43.915337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:43.915287 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:25:43.921449 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:43.921418 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc"] Oct 31 00:25:43.965072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:43.965038 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:25:43.971659 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:43.971635 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba5bd7ac-zncsc\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:43.971745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:43.971684 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbkhn\" (UniqueName: \"kubernetes.io/projected/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-kube-api-access-jbkhn\") pod \"maas-api-revoke-sub-ba5bd7ac-zncsc\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:43.971850 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:43.971833 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba5bd7ac-zncsc\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:44.073012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.072982 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba5bd7ac-zncsc\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:44.073161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.073028 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbkhn\" (UniqueName: \"kubernetes.io/projected/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-kube-api-access-jbkhn\") pod \"maas-api-revoke-sub-ba5bd7ac-zncsc\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:44.073161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.073105 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba5bd7ac-zncsc\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:44.073724 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.073700 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba5bd7ac-zncsc\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:44.075441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.075424 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba5bd7ac-zncsc\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:44.134028 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.133999 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbkhn\" (UniqueName: \"kubernetes.io/projected/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-kube-api-access-jbkhn\") pod \"maas-api-revoke-sub-ba5bd7ac-zncsc\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:44.177025 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.176999 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:44.555820 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:25:44.555790 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe76a6b9_2442_4bdf_bf86_ed3ba919a6e5.slice/crio-d11add1bcf4d4c5314d84d0ddaf0f6af74d8a56611f359977878e934e2df6e85 WatchSource:0}: Error finding container d11add1bcf4d4c5314d84d0ddaf0f6af74d8a56611f359977878e934e2df6e85: Status 404 returned error can't find the container with id d11add1bcf4d4c5314d84d0ddaf0f6af74d8a56611f359977878e934e2df6e85 Oct 31 00:25:44.556306 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.556283 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc"] Oct 31 00:25:44.939125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.939093 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" event={"ID":"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5","Type":"ContainerStarted","Data":"23e1c219c1196936966a4b8577afde9e4d653342b2da8fafc1051b999db407bc"} Oct 31 00:25:44.939520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:44.939151 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" event={"ID":"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5","Type":"ContainerStarted","Data":"d11add1bcf4d4c5314d84d0ddaf0f6af74d8a56611f359977878e934e2df6e85"} Oct 31 00:25:45.057704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:45.057603 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" podStartSLOduration=2.057576469 podStartE2EDuration="2.057576469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:45.053887133 +0000 UTC m=+1377.529245558" watchObservedRunningTime="2026-10-31 00:25:45.057576469 +0000 UTC m=+1377.532934893" Oct 31 00:25:45.952694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:45.952653 4969 generic.go:356] "Generic (PLEG): container finished" podID="be76a6b9-2442-4bdf-bf86-ed3ba919a6e5" containerID="23e1c219c1196936966a4b8577afde9e4d653342b2da8fafc1051b999db407bc" exitCode=0 Oct 31 00:25:45.953198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:45.952849 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" event={"ID":"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5","Type":"ContainerDied","Data":"23e1c219c1196936966a4b8577afde9e4d653342b2da8fafc1051b999db407bc"} Oct 31 00:25:47.120054 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.120030 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:47.208101 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.208066 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-xks-ca\") pod \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " Oct 31 00:25:47.208292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.208148 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-openshift-ca\") pod \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " Oct 31 00:25:47.208292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.208198 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-kube-api-access-jbkhn\" (UniqueName: \"kubernetes.io/projected/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-kube-api-access-jbkhn\") pod \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\" (UID: \"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5\") " Oct 31 00:25:47.208500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.208471 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-openshift-ca" pod "be76a6b9-2442-4bdf-bf86-ed3ba919a6e5" (UID: "be76a6b9-2442-4bdf-bf86-ed3ba919a6e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:47.208680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.208664 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:47.210296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.210271 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-xks-ca" pod "be76a6b9-2442-4bdf-bf86-ed3ba919a6e5" (UID: "be76a6b9-2442-4bdf-bf86-ed3ba919a6e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:47.210637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.210572 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-kube-api-access-jbkhn" pod "be76a6b9-2442-4bdf-bf86-ed3ba919a6e5" (UID: "be76a6b9-2442-4bdf-bf86-ed3ba919a6e5"). InnerVolumeSpecName "kube-api-access-jbkhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:47.309340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.309309 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:47.309340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.309335 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbkhn\" (UniqueName: \"kubernetes.io/projected/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5-kube-api-access-jbkhn\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:47.950161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.950123 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw"] Oct 31 00:25:47.950777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.950758 4969 cpu_manager.go:414] "containerMap: removing container" podUID="be76a6b9-2442-4bdf-bf86-ed3ba919a6e5" containerName="revoke-keys" Oct 31 00:25:47.950898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.950778 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be76a6b9-2442-4bdf-bf86-ed3ba919a6e5" containerName="revoke-keys" Oct 31 00:25:47.950956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.950926 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="be76a6b9-2442-4bdf-bf86-ed3ba919a6e5" containerName="revoke-keys" Oct 31 00:25:47.954445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.954424 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:47.968892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.968873 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" Oct 31 00:25:47.968992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.968911 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc" event={"ID":"be76a6b9-2442-4bdf-bf86-ed3ba919a6e5","Type":"ContainerDied","Data":"d11add1bcf4d4c5314d84d0ddaf0f6af74d8a56611f359977878e934e2df6e85"} Oct 31 00:25:47.968992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:47.968948 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d11add1bcf4d4c5314d84d0ddaf0f6af74d8a56611f359977878e934e2df6e85" Oct 31 00:25:48.015197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.015117 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw"] Oct 31 00:25:48.017930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.017900 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfnr6\" (UniqueName: \"kubernetes.io/projected/b37c7b3b-c3b5-42a0-97b7-55827c224955-kube-api-access-sfnr6\") pod \"maas-api-revoke-sub-98daaf6f-zzpqw\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.018108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.018019 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98daaf6f-zzpqw\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.018180 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.018107 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98daaf6f-zzpqw\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.119158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.119125 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfnr6\" (UniqueName: \"kubernetes.io/projected/b37c7b3b-c3b5-42a0-97b7-55827c224955-kube-api-access-sfnr6\") pod \"maas-api-revoke-sub-98daaf6f-zzpqw\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.119402 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.119208 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98daaf6f-zzpqw\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.119402 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.119260 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98daaf6f-zzpqw\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.119939 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.119917 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98daaf6f-zzpqw\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.121843 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.121818 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98daaf6f-zzpqw\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.180163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.180135 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfnr6\" (UniqueName: \"kubernetes.io/projected/b37c7b3b-c3b5-42a0-97b7-55827c224955-kube-api-access-sfnr6\") pod \"maas-api-revoke-sub-98daaf6f-zzpqw\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.268369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.268294 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:48.650277 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:25:48.650247 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37c7b3b_c3b5_42a0_97b7_55827c224955.slice/crio-56d746db7ee70c9cf8fb2a7171f5364919a7d46188af97cd358cfae780ad98a1 WatchSource:0}: Error finding container 56d746db7ee70c9cf8fb2a7171f5364919a7d46188af97cd358cfae780ad98a1: Status 404 returned error can't find the container with id 56d746db7ee70c9cf8fb2a7171f5364919a7d46188af97cd358cfae780ad98a1 Oct 31 00:25:48.652404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.652375 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw"] Oct 31 00:25:48.977625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:48.977522 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" event={"ID":"b37c7b3b-c3b5-42a0-97b7-55827c224955","Type":"ContainerStarted","Data":"56d746db7ee70c9cf8fb2a7171f5364919a7d46188af97cd358cfae780ad98a1"} Oct 31 00:25:49.953304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:49.953265 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:49.965904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:49.965876 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:49.986354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:49.986327 4969 generic.go:356] "Generic (PLEG): container finished" podID="b37c7b3b-c3b5-42a0-97b7-55827c224955" containerID="823fd5237d79f6b52a1c0a7965dfbd8cb21ddb1e0596c79fd30bc2f84e710242" exitCode=0 Oct 31 00:25:49.986476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:49.986406 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" event={"ID":"b37c7b3b-c3b5-42a0-97b7-55827c224955","Type":"ContainerDied","Data":"823fd5237d79f6b52a1c0a7965dfbd8cb21ddb1e0596c79fd30bc2f84e710242"} Oct 31 00:25:49.998515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:49.998497 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:25:50.072350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:50.072314 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs"] Oct 31 00:25:50.075867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:50.075847 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f688568f-5rzrs"] Oct 31 00:25:51.147963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.147940 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:51.259197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.259113 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-openshift-ca\") pod \"b37c7b3b-c3b5-42a0-97b7-55827c224955\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " Oct 31 00:25:51.259197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.259143 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-xks-ca\") pod \"b37c7b3b-c3b5-42a0-97b7-55827c224955\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " Oct 31 00:25:51.259197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.259189 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37c7b3b-c3b5-42a0-97b7-55827c224955-kube-api-access-sfnr6\" (UniqueName: \"kubernetes.io/projected/b37c7b3b-c3b5-42a0-97b7-55827c224955-kube-api-access-sfnr6\") pod \"b37c7b3b-c3b5-42a0-97b7-55827c224955\" (UID: \"b37c7b3b-c3b5-42a0-97b7-55827c224955\") " Oct 31 00:25:51.259487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.259459 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-openshift-ca" pod "b37c7b3b-c3b5-42a0-97b7-55827c224955" (UID: "b37c7b3b-c3b5-42a0-97b7-55827c224955"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:51.259722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.259705 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:51.261498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.261469 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-xks-ca" pod "b37c7b3b-c3b5-42a0-97b7-55827c224955" (UID: "b37c7b3b-c3b5-42a0-97b7-55827c224955"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:51.261678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.261510 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37c7b3b-c3b5-42a0-97b7-55827c224955-kube-api-access-sfnr6" pod "b37c7b3b-c3b5-42a0-97b7-55827c224955" (UID: "b37c7b3b-c3b5-42a0-97b7-55827c224955"). InnerVolumeSpecName "kube-api-access-sfnr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:51.361285 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.361248 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37c7b3b-c3b5-42a0-97b7-55827c224955-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:51.361285 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.361280 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfnr6\" (UniqueName: \"kubernetes.io/projected/b37c7b3b-c3b5-42a0-97b7-55827c224955-kube-api-access-sfnr6\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:25:51.734714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:51.734682 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c3b141-91f6-42c2-a388-5daf4ac59b87" path="/var/lib/kubelet/pods/d3c3b141-91f6-42c2-a388-5daf4ac59b87/volumes" Oct 31 00:25:52.003419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:52.003334 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" Oct 31 00:25:52.003419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:52.003368 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw" event={"ID":"b37c7b3b-c3b5-42a0-97b7-55827c224955","Type":"ContainerDied","Data":"56d746db7ee70c9cf8fb2a7171f5364919a7d46188af97cd358cfae780ad98a1"} Oct 31 00:25:52.003619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:52.003425 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56d746db7ee70c9cf8fb2a7171f5364919a7d46188af97cd358cfae780ad98a1" Oct 31 00:25:58.043489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.043456 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8"] Oct 31 00:25:58.044020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.044006 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b37c7b3b-c3b5-42a0-97b7-55827c224955" containerName="revoke-keys" Oct 31 00:25:58.044064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.044020 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37c7b3b-c3b5-42a0-97b7-55827c224955" containerName="revoke-keys" Oct 31 00:25:58.044147 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.044138 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37c7b3b-c3b5-42a0-97b7-55827c224955" containerName="revoke-keys" Oct 31 00:25:58.048835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.048819 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.105503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.105464 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8"] Oct 31 00:25:58.149863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.149831 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:25:58.150030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.149874 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:25:58.163136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.163101 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btg9w\" (UniqueName: \"kubernetes.io/projected/417345c2-7500-4548-9057-f9fbb5081037-kube-api-access-btg9w\") pod \"maas-api-revoke-sub-c9393643-5q2d8\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.163295 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.163219 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417345c2-7500-4548-9057-f9fbb5081037-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9393643-5q2d8\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.163365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.163303 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417345c2-7500-4548-9057-f9fbb5081037-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9393643-5q2d8\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.266611 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.266483 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btg9w\" (UniqueName: \"kubernetes.io/projected/417345c2-7500-4548-9057-f9fbb5081037-kube-api-access-btg9w\") pod \"maas-api-revoke-sub-c9393643-5q2d8\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.266611 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.266561 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417345c2-7500-4548-9057-f9fbb5081037-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9393643-5q2d8\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.266884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.266614 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417345c2-7500-4548-9057-f9fbb5081037-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9393643-5q2d8\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.267616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.267576 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417345c2-7500-4548-9057-f9fbb5081037-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9393643-5q2d8\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.270001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.269974 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417345c2-7500-4548-9057-f9fbb5081037-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9393643-5q2d8\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.328470 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.328398 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btg9w\" (UniqueName: \"kubernetes.io/projected/417345c2-7500-4548-9057-f9fbb5081037-kube-api-access-btg9w\") pod \"maas-api-revoke-sub-c9393643-5q2d8\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.362451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.362420 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:25:58.743624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:58.743567 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8"] Oct 31 00:25:58.743801 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:25:58.743777 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417345c2_7500_4548_9057_f9fbb5081037.slice/crio-389a38777661eec4ade4c64b9d86a819f23b32a12ff65066ba55b684e4e847dc WatchSource:0}: Error finding container 389a38777661eec4ade4c64b9d86a819f23b32a12ff65066ba55b684e4e847dc: Status 404 returned error can't find the container with id 389a38777661eec4ade4c64b9d86a819f23b32a12ff65066ba55b684e4e847dc Oct 31 00:25:59.085229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:25:59.085197 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" event={"ID":"417345c2-7500-4548-9057-f9fbb5081037","Type":"ContainerStarted","Data":"389a38777661eec4ade4c64b9d86a819f23b32a12ff65066ba55b684e4e847dc"} Oct 31 00:26:00.096513 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:00.096425 4969 generic.go:356] "Generic (PLEG): container finished" podID="417345c2-7500-4548-9057-f9fbb5081037" containerID="c707725cc214ae65cb47aae5a4dbe3eb93ddeb0f0d467b8f5a5dba6bc22fa51e" exitCode=0 Oct 31 00:26:00.096960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:00.096507 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" event={"ID":"417345c2-7500-4548-9057-f9fbb5081037","Type":"ContainerDied","Data":"c707725cc214ae65cb47aae5a4dbe3eb93ddeb0f0d467b8f5a5dba6bc22fa51e"} Oct 31 00:26:01.263746 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.263715 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:26:01.303260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.303226 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417345c2-7500-4548-9057-f9fbb5081037-kube-api-access-btg9w\" (UniqueName: \"kubernetes.io/projected/417345c2-7500-4548-9057-f9fbb5081037-kube-api-access-btg9w\") pod \"417345c2-7500-4548-9057-f9fbb5081037\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " Oct 31 00:26:01.303435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.303276 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417345c2-7500-4548-9057-f9fbb5081037-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417345c2-7500-4548-9057-f9fbb5081037-maas-api-xks-ca\") pod \"417345c2-7500-4548-9057-f9fbb5081037\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " Oct 31 00:26:01.303435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.303320 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417345c2-7500-4548-9057-f9fbb5081037-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417345c2-7500-4548-9057-f9fbb5081037-maas-api-openshift-ca\") pod \"417345c2-7500-4548-9057-f9fbb5081037\" (UID: \"417345c2-7500-4548-9057-f9fbb5081037\") " Oct 31 00:26:01.303758 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.303736 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417345c2-7500-4548-9057-f9fbb5081037-maas-api-openshift-ca" pod "417345c2-7500-4548-9057-f9fbb5081037" (UID: "417345c2-7500-4548-9057-f9fbb5081037"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:01.303857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.303838 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417345c2-7500-4548-9057-f9fbb5081037-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:01.305648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.305618 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417345c2-7500-4548-9057-f9fbb5081037-kube-api-access-btg9w" pod "417345c2-7500-4548-9057-f9fbb5081037" (UID: "417345c2-7500-4548-9057-f9fbb5081037"). InnerVolumeSpecName "kube-api-access-btg9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:01.305648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.305638 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417345c2-7500-4548-9057-f9fbb5081037-maas-api-xks-ca" pod "417345c2-7500-4548-9057-f9fbb5081037" (UID: "417345c2-7500-4548-9057-f9fbb5081037"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:01.405179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.405141 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btg9w\" (UniqueName: \"kubernetes.io/projected/417345c2-7500-4548-9057-f9fbb5081037-kube-api-access-btg9w\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:01.405179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.405170 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417345c2-7500-4548-9057-f9fbb5081037-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:01.675243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.674234 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns"] Oct 31 00:26:01.676958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.675884 4969 cpu_manager.go:414] "containerMap: removing container" podUID="417345c2-7500-4548-9057-f9fbb5081037" containerName="revoke-keys" Oct 31 00:26:01.676958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.675909 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417345c2-7500-4548-9057-f9fbb5081037" containerName="revoke-keys" Oct 31 00:26:01.676958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.676220 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="417345c2-7500-4548-9057-f9fbb5081037" containerName="revoke-keys" Oct 31 00:26:01.684101 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.684067 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.737222 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.737193 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns"] Oct 31 00:26:01.808416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.808373 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2846a30b-9rhns\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.808665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.808484 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc57r\" (UniqueName: \"kubernetes.io/projected/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-kube-api-access-zc57r\") pod \"maas-api-revoke-sub-2846a30b-9rhns\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.808719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.808692 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2846a30b-9rhns\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.909780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.909743 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2846a30b-9rhns\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.909968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.909845 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2846a30b-9rhns\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.909968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.909911 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc57r\" (UniqueName: \"kubernetes.io/projected/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-kube-api-access-zc57r\") pod \"maas-api-revoke-sub-2846a30b-9rhns\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.910529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.910507 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2846a30b-9rhns\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.912379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.912356 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2846a30b-9rhns\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.971573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.971505 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc57r\" (UniqueName: \"kubernetes.io/projected/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-kube-api-access-zc57r\") pod \"maas-api-revoke-sub-2846a30b-9rhns\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:01.998354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:01.998325 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:02.114077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.114043 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" Oct 31 00:26:02.114272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.114082 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8" event={"ID":"417345c2-7500-4548-9057-f9fbb5081037","Type":"ContainerDied","Data":"389a38777661eec4ade4c64b9d86a819f23b32a12ff65066ba55b684e4e847dc"} Oct 31 00:26:02.114272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.114145 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="389a38777661eec4ade4c64b9d86a819f23b32a12ff65066ba55b684e4e847dc" Oct 31 00:26:02.289373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.289293 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h"] Oct 31 00:26:02.294622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.294585 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.399261 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.399228 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h"] Oct 31 00:26:02.407810 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:02.407758 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13162a1a_fa05_44b7_9ec5_83dd9cd32f4d.slice/crio-e99acabb8b1e6c3984eae771cfcaeeacd594909e9986744943ac80bf789fb93a WatchSource:0}: Error finding container e99acabb8b1e6c3984eae771cfcaeeacd594909e9986744943ac80bf789fb93a: Status 404 returned error can't find the container with id e99acabb8b1e6c3984eae771cfcaeeacd594909e9986744943ac80bf789fb93a Oct 31 00:26:02.419244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.419206 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.419371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.419279 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632906a4-9437-49ae-b08c-a67e11cb6cb7-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.419371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.419314 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.419371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.419358 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.419496 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.419403 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pc4z\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-kube-api-access-9pc4z\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.419530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.419510 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.419564 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.419551 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.419714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.419698 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.419779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.419766 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.422946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.422924 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns"] Oct 31 00:26:02.520996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.520961 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pc4z\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-kube-api-access-9pc4z\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.521199 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521086 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.521199 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521135 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.521199 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521165 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.521370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521216 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.521370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521316 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.521370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521362 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632906a4-9437-49ae-b08c-a67e11cb6cb7-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.521521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521395 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.521521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521443 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.521648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521534 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.522664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.521937 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.522664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.522021 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.522664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.522232 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632906a4-9437-49ae-b08c-a67e11cb6cb7-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.522664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.522450 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.524128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.524100 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.524305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.524283 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.582110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.582018 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.582265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.582134 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pc4z\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-kube-api-access-9pc4z\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.609746 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:02.609711 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:02.764181 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:02.764149 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:26:03.042021 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:03.041990 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632906a4_9437_49ae_b08c_a67e11cb6cb7.slice/crio-df54032cc363173c51407a2c9d0fefec471534b1ff7d00df64801facd1bd48bc WatchSource:0}: Error finding container df54032cc363173c51407a2c9d0fefec471534b1ff7d00df64801facd1bd48bc: Status 404 returned error can't find the container with id df54032cc363173c51407a2c9d0fefec471534b1ff7d00df64801facd1bd48bc Oct 31 00:26:03.042118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.042060 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h"] Oct 31 00:26:03.044403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.044375 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:03.044482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.044438 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:03.044482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.044467 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:03.123758 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.123725 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" event={"ID":"632906a4-9437-49ae-b08c-a67e11cb6cb7","Type":"ContainerStarted","Data":"3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735"} Oct 31 00:26:03.123890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.123789 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" event={"ID":"632906a4-9437-49ae-b08c-a67e11cb6cb7","Type":"ContainerStarted","Data":"df54032cc363173c51407a2c9d0fefec471534b1ff7d00df64801facd1bd48bc"} Oct 31 00:26:03.125945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.125921 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" event={"ID":"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d","Type":"ContainerStarted","Data":"52e57bd9112aae4d3df264c96c0cb9717952c645f2326065d8b54fd17f477279"} Oct 31 00:26:03.126068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.125974 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" event={"ID":"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d","Type":"ContainerStarted","Data":"e99acabb8b1e6c3984eae771cfcaeeacd594909e9986744943ac80bf789fb93a"} Oct 31 00:26:03.251138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.251079 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" podStartSLOduration=1.251060246 podStartE2EDuration="1.251060246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:03.246190337 +0000 UTC m=+1395.721548755" watchObservedRunningTime="2026-10-31 00:26:03.251060246 +0000 UTC m=+1395.726418672" Oct 31 00:26:03.374754 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.374702 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" podStartSLOduration=2.374687791 podStartE2EDuration="2.374687791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:03.37080124 +0000 UTC m=+1395.846159663" watchObservedRunningTime="2026-10-31 00:26:03.374687791 +0000 UTC m=+1395.850046214" Oct 31 00:26:03.609889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:03.609791 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:04.138703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:04.138671 4969 generic.go:356] "Generic (PLEG): container finished" podID="13162a1a-fa05-44b7-9ec5-83dd9cd32f4d" containerID="52e57bd9112aae4d3df264c96c0cb9717952c645f2326065d8b54fd17f477279" exitCode=0 Oct 31 00:26:04.138859 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:04.138741 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" event={"ID":"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d","Type":"ContainerDied","Data":"52e57bd9112aae4d3df264c96c0cb9717952c645f2326065d8b54fd17f477279"} Oct 31 00:26:04.612110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:04.612015 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:04.612110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:04.612081 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:05.348979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.348948 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:05.462488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.462405 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-kube-api-access-zc57r\" (UniqueName: \"kubernetes.io/projected/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-kube-api-access-zc57r\") pod \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " Oct 31 00:26:05.462663 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.462628 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-xks-ca\") pod \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " Oct 31 00:26:05.462715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.462689 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-openshift-ca\") pod \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\" (UID: \"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d\") " Oct 31 00:26:05.463044 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.463002 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-openshift-ca" pod "13162a1a-fa05-44b7-9ec5-83dd9cd32f4d" (UID: "13162a1a-fa05-44b7-9ec5-83dd9cd32f4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:05.463181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.463163 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:05.464659 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.464637 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-kube-api-access-zc57r" pod "13162a1a-fa05-44b7-9ec5-83dd9cd32f4d" (UID: "13162a1a-fa05-44b7-9ec5-83dd9cd32f4d"). InnerVolumeSpecName "kube-api-access-zc57r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:05.464831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.464810 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-xks-ca" pod "13162a1a-fa05-44b7-9ec5-83dd9cd32f4d" (UID: "13162a1a-fa05-44b7-9ec5-83dd9cd32f4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:05.564546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.564516 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:05.564546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.564542 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc57r\" (UniqueName: \"kubernetes.io/projected/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d-kube-api-access-zc57r\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:05.610096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.610060 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:05.610248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:05.610117 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:06.154756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:06.154726 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" Oct 31 00:26:06.155247 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:06.154752 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns" event={"ID":"13162a1a-fa05-44b7-9ec5-83dd9cd32f4d","Type":"ContainerDied","Data":"e99acabb8b1e6c3984eae771cfcaeeacd594909e9986744943ac80bf789fb93a"} Oct 31 00:26:06.155247 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:06.154793 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99acabb8b1e6c3984eae771cfcaeeacd594909e9986744943ac80bf789fb93a" Oct 31 00:26:06.610517 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:06.610428 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:06.610517 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:06.610486 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:06.783620 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:06.776780 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-282885dac35a: secret "agc-cred-282885dac35a" not found Oct 31 00:26:06.783620 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:06.776816 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-43bfee711d1b: secret "agc-cred-43bfee711d1b" not found Oct 31 00:26:06.783620 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:06.776835 4969 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6: [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found] Oct 31 00:26:06.783620 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:06.776919 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials podName:f598e6e3-f02b-4750-b717-ff5ecba6477c nodeName:}" failed. No retries permitted until 2026-10-31 00:26:07.276886681 +0000 UTC m=+1399.752245101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials") pod "payload-processing-external-model-657c7b7984-zkcf6" (UID: "f598e6e3-f02b-4750-b717-ff5ecba6477c") : [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found] Oct 31 00:26:07.303853 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:07.303817 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-282885dac35a: secret "agc-cred-282885dac35a" not found Oct 31 00:26:07.303853 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:07.303848 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-43bfee711d1b: secret "agc-cred-43bfee711d1b" not found Oct 31 00:26:07.303853 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:07.303863 4969 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6: [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found] Oct 31 00:26:07.304483 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:07.303923 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials podName:f598e6e3-f02b-4750-b717-ff5ecba6477c nodeName:}" failed. No retries permitted until 2026-10-31 00:26:08.303908482 +0000 UTC m=+1400.779266884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials") pod "payload-processing-external-model-657c7b7984-zkcf6" (UID: "f598e6e3-f02b-4750-b717-ff5ecba6477c") : [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found] Oct 31 00:26:07.610319 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:07.610234 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:07.610319 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:07.610287 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:08.316342 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:08.316308 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-282885dac35a: secret "agc-cred-282885dac35a" not found Oct 31 00:26:08.316342 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:08.316335 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-43bfee711d1b: secret "agc-cred-43bfee711d1b" not found Oct 31 00:26:08.316342 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:08.316348 4969 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6: [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found] Oct 31 00:26:08.316849 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:08.316402 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials podName:f598e6e3-f02b-4750-b717-ff5ecba6477c nodeName:}" failed. No retries permitted until 2026-10-31 00:26:10.316385941 +0000 UTC m=+1402.791744343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials") pod "payload-processing-external-model-657c7b7984-zkcf6" (UID: "f598e6e3-f02b-4750-b717-ff5ecba6477c") : [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found] Oct 31 00:26:08.610607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:08.610505 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:08.610607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:08.610561 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:09.610226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:09.610190 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:09.610800 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:09.610244 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:10.348659 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:10.344688 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-282885dac35a: secret "agc-cred-282885dac35a" not found Oct 31 00:26:10.348659 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:10.344724 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-43bfee711d1b: secret "agc-cred-43bfee711d1b" not found Oct 31 00:26:10.348659 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:10.344739 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a2f1ffbe7a37: secret "agc-cred-a2f1ffbe7a37" not found Oct 31 00:26:10.348659 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:10.344752 4969 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6: [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found, secret "agc-cred-a2f1ffbe7a37" not found] Oct 31 00:26:10.348659 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:10.344821 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials podName:f598e6e3-f02b-4750-b717-ff5ecba6477c nodeName:}" failed. No retries permitted until 2026-10-31 00:26:14.344800485 +0000 UTC m=+1406.820158905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials") pod "payload-processing-external-model-657c7b7984-zkcf6" (UID: "f598e6e3-f02b-4750-b717-ff5ecba6477c") : [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found, secret "agc-cred-a2f1ffbe7a37" not found] Oct 31 00:26:10.443027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:10.442995 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6"] Oct 31 00:26:10.615256 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:10.615231 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:10.615740 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:10.615616 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:10.616307 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:10.616288 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:26:12.480265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.480235 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4"] Oct 31 00:26:12.480824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.480810 4969 cpu_manager.go:414] "containerMap: removing container" podUID="13162a1a-fa05-44b7-9ec5-83dd9cd32f4d" containerName="revoke-keys" Oct 31 00:26:12.480871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.480825 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13162a1a-fa05-44b7-9ec5-83dd9cd32f4d" containerName="revoke-keys" Oct 31 00:26:12.480951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.480942 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="13162a1a-fa05-44b7-9ec5-83dd9cd32f4d" containerName="revoke-keys" Oct 31 00:26:12.484713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.484696 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.543142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.543109 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4"] Oct 31 00:26:12.570797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.570758 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgzm\" (UniqueName: \"kubernetes.io/projected/f99c713e-9226-4222-acee-41a5179b29d6-kube-api-access-9bgzm\") pod \"maas-api-revoke-sub-48e19166-gqvd4\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.570979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.570890 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e19166-gqvd4\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.571043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.571028 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e19166-gqvd4\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.586259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.586228 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:12.586392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.586273 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:26:12.672018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.671979 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e19166-gqvd4\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.672183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.672108 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e19166-gqvd4\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.672257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.672243 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgzm\" (UniqueName: \"kubernetes.io/projected/f99c713e-9226-4222-acee-41a5179b29d6-kube-api-access-9bgzm\") pod \"maas-api-revoke-sub-48e19166-gqvd4\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.672847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.672791 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e19166-gqvd4\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.674604 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.674568 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e19166-gqvd4\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.733165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.733098 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgzm\" (UniqueName: \"kubernetes.io/projected/f99c713e-9226-4222-acee-41a5179b29d6-kube-api-access-9bgzm\") pod \"maas-api-revoke-sub-48e19166-gqvd4\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:12.798983 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:12.798951 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:13.180052 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:13.180024 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4"] Oct 31 00:26:13.180391 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:13.180371 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99c713e_9226_4222_acee_41a5179b29d6.slice/crio-a3a5860b78d21ac680503a64698c9c800c336f36e53cbe58ecdc3cf4e842795c WatchSource:0}: Error finding container a3a5860b78d21ac680503a64698c9c800c336f36e53cbe58ecdc3cf4e842795c: Status 404 returned error can't find the container with id a3a5860b78d21ac680503a64698c9c800c336f36e53cbe58ecdc3cf4e842795c Oct 31 00:26:13.208851 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:13.208822 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" event={"ID":"f99c713e-9226-4222-acee-41a5179b29d6","Type":"ContainerStarted","Data":"a3a5860b78d21ac680503a64698c9c800c336f36e53cbe58ecdc3cf4e842795c"} Oct 31 00:26:14.218587 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:14.218555 4969 generic.go:356] "Generic (PLEG): container finished" podID="f99c713e-9226-4222-acee-41a5179b29d6" containerID="36f9b8902f9497cee9c2f9a656c48355418f49dd41752b39b19c185f1bec2bb6" exitCode=0 Oct 31 00:26:14.218998 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:14.218644 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" event={"ID":"f99c713e-9226-4222-acee-41a5179b29d6","Type":"ContainerDied","Data":"36f9b8902f9497cee9c2f9a656c48355418f49dd41752b39b19c185f1bec2bb6"} Oct 31 00:26:14.393353 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:14.393316 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-282885dac35a: secret "agc-cred-282885dac35a" not found Oct 31 00:26:14.393353 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:14.393343 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-43bfee711d1b: secret "agc-cred-43bfee711d1b" not found Oct 31 00:26:14.393353 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:14.393349 4969 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a2f1ffbe7a37: secret "agc-cred-a2f1ffbe7a37" not found Oct 31 00:26:14.393353 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:14.393358 4969 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6: [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found, secret "agc-cred-a2f1ffbe7a37" not found] Oct 31 00:26:14.393679 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:14.393423 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials podName:f598e6e3-f02b-4750-b717-ff5ecba6477c nodeName:}" failed. No retries permitted until 2026-10-31 00:26:22.393408916 +0000 UTC m=+1414.868767331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials") pod "payload-processing-external-model-657c7b7984-zkcf6" (UID: "f598e6e3-f02b-4750-b717-ff5ecba6477c") : [secret "agc-cred-282885dac35a" not found, secret "agc-cred-43bfee711d1b" not found, secret "agc-cred-a2f1ffbe7a37" not found] Oct 31 00:26:15.387811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.387787 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:15.444240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.444206 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" podUID="f598e6e3-f02b-4750-b717-ff5ecba6477c" containerName="payload-processing" containerID="cri-o://f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7" gracePeriod=25 Oct 31 00:26:15.504867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.504776 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99c713e-9226-4222-acee-41a5179b29d6-kube-api-access-9bgzm\" (UniqueName: \"kubernetes.io/projected/f99c713e-9226-4222-acee-41a5179b29d6-kube-api-access-9bgzm\") pod \"f99c713e-9226-4222-acee-41a5179b29d6\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " Oct 31 00:26:15.506583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.504950 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-xks-ca\") pod \"f99c713e-9226-4222-acee-41a5179b29d6\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " Oct 31 00:26:15.506583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.505064 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-openshift-ca\") pod \"f99c713e-9226-4222-acee-41a5179b29d6\" (UID: \"f99c713e-9226-4222-acee-41a5179b29d6\") " Oct 31 00:26:15.506583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.505404 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-openshift-ca" pod "f99c713e-9226-4222-acee-41a5179b29d6" (UID: "f99c713e-9226-4222-acee-41a5179b29d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:15.506583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.505665 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.507729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.507688 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-xks-ca" pod "f99c713e-9226-4222-acee-41a5179b29d6" (UID: "f99c713e-9226-4222-acee-41a5179b29d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:15.507729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.507711 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99c713e-9226-4222-acee-41a5179b29d6-kube-api-access-9bgzm" pod "f99c713e-9226-4222-acee-41a5179b29d6" (UID: "f99c713e-9226-4222-acee-41a5179b29d6"). InnerVolumeSpecName "kube-api-access-9bgzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:15.607308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.607270 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgzm\" (UniqueName: \"kubernetes.io/projected/f99c713e-9226-4222-acee-41a5179b29d6-kube-api-access-9bgzm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.607308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.607300 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c713e-9226-4222-acee-41a5179b29d6-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.701181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.701159 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:26:15.809641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.809548 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-plugins-config-volume\") pod \"f598e6e3-f02b-4750-b717-ff5ecba6477c\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " Oct 31 00:26:15.809641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.809633 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials\") pod \"f598e6e3-f02b-4750-b717-ff5ecba6477c\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " Oct 31 00:26:15.809811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.809731 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-kube-api-access-b69bf\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-kube-api-access-b69bf\") pod \"f598e6e3-f02b-4750-b717-ff5ecba6477c\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " Oct 31 00:26:15.809811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.809758 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-routing-overlay-volume\") pod \"f598e6e3-f02b-4750-b717-ff5ecba6477c\" (UID: \"f598e6e3-f02b-4750-b717-ff5ecba6477c\") " Oct 31 00:26:15.809943 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.809920 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-plugins-config-volume" pod "f598e6e3-f02b-4750-b717-ff5ecba6477c" (UID: "f598e6e3-f02b-4750-b717-ff5ecba6477c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:15.810167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.810144 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-routing-overlay-volume" pod "f598e6e3-f02b-4750-b717-ff5ecba6477c" (UID: "f598e6e3-f02b-4750-b717-ff5ecba6477c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:15.810245 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.810232 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.810297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.810247 4969 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f598e6e3-f02b-4750-b717-ff5ecba6477c-routing-overlay-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.811863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.811843 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials" pod "f598e6e3-f02b-4750-b717-ff5ecba6477c" (UID: "f598e6e3-f02b-4750-b717-ff5ecba6477c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:15.812100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.812082 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-kube-api-access-b69bf" pod "f598e6e3-f02b-4750-b717-ff5ecba6477c" (UID: "f598e6e3-f02b-4750-b717-ff5ecba6477c"). InnerVolumeSpecName "kube-api-access-b69bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:15.911403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.911361 4969 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-provider-credentials\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.911403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:15.911406 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b69bf\" (UniqueName: \"kubernetes.io/projected/f598e6e3-f02b-4750-b717-ff5ecba6477c-kube-api-access-b69bf\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:16.048329 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.048292 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v"] Oct 31 00:26:16.048922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.048901 4969 cpu_manager.go:414] "containerMap: removing container" podUID="f598e6e3-f02b-4750-b717-ff5ecba6477c" containerName="payload-processing" Oct 31 00:26:16.048922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.048916 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f598e6e3-f02b-4750-b717-ff5ecba6477c" containerName="payload-processing" Oct 31 00:26:16.049158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.048951 4969 cpu_manager.go:414] "containerMap: removing container" podUID="f99c713e-9226-4222-acee-41a5179b29d6" containerName="revoke-keys" Oct 31 00:26:16.049158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.048956 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99c713e-9226-4222-acee-41a5179b29d6" containerName="revoke-keys" Oct 31 00:26:16.049158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.049042 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99c713e-9226-4222-acee-41a5179b29d6" containerName="revoke-keys" Oct 31 00:26:16.049158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.049054 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="f598e6e3-f02b-4750-b717-ff5ecba6477c" containerName="payload-processing" Oct 31 00:26:16.052824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.052807 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.109965 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.109932 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v"] Oct 31 00:26:16.114008 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.113974 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a7cfdf-4sb8v\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.114159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.114095 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hx4\" (UniqueName: \"kubernetes.io/projected/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-kube-api-access-c2hx4\") pod \"maas-api-revoke-sub-39a7cfdf-4sb8v\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.114232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.114211 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a7cfdf-4sb8v\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.214988 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.214958 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a7cfdf-4sb8v\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.215172 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.215060 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a7cfdf-4sb8v\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.215172 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.215143 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hx4\" (UniqueName: \"kubernetes.io/projected/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-kube-api-access-c2hx4\") pod \"maas-api-revoke-sub-39a7cfdf-4sb8v\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.215792 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.215771 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a7cfdf-4sb8v\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.217631 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.217613 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a7cfdf-4sb8v\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.236783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.236756 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" Oct 31 00:26:16.236938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.236796 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4" event={"ID":"f99c713e-9226-4222-acee-41a5179b29d6","Type":"ContainerDied","Data":"a3a5860b78d21ac680503a64698c9c800c336f36e53cbe58ecdc3cf4e842795c"} Oct 31 00:26:16.236938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.236846 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a5860b78d21ac680503a64698c9c800c336f36e53cbe58ecdc3cf4e842795c" Oct 31 00:26:16.238350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.238326 4969 generic.go:356] "Generic (PLEG): container finished" podID="f598e6e3-f02b-4750-b717-ff5ecba6477c" containerID="f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7" exitCode=0 Oct 31 00:26:16.238447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.238367 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" event={"ID":"f598e6e3-f02b-4750-b717-ff5ecba6477c","Type":"ContainerDied","Data":"f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7"} Oct 31 00:26:16.238447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.238418 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" event={"ID":"f598e6e3-f02b-4750-b717-ff5ecba6477c","Type":"ContainerDied","Data":"684497b76a2e8719c3152fa6feb94baedf8370e6a553d5db58bb35669d5d0e8c"} Oct 31 00:26:16.238520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.238374 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6" Oct 31 00:26:16.238520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.238448 4969 scope.go:122] "RemoveContainer" containerID="f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7" Oct 31 00:26:16.249871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.249857 4969 scope.go:122] "RemoveContainer" containerID="f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7" Oct 31 00:26:16.250132 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:16.250113 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7\": container with ID starting with f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7 not found: ID does not exist" containerID="f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7" Oct 31 00:26:16.250192 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.250138 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7"} err="failed to get container status \"f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7\": rpc error: code = NotFound desc = could not find container \"f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7\": container with ID starting with f41e73bd357d3d4dc95c6c8a1a825ff9d7ea1c26febd23a0c7797c7d7a4660c7 not found: ID does not exist" Oct 31 00:26:16.277646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.277627 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hx4\" (UniqueName: \"kubernetes.io/projected/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-kube-api-access-c2hx4\") pod \"maas-api-revoke-sub-39a7cfdf-4sb8v\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.366649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.366584 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:16.460069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.460039 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6"] Oct 31 00:26:16.464344 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.464321 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-657c7b7984-zkcf6"] Oct 31 00:26:16.750971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:16.750868 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v"] Oct 31 00:26:16.751264 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:16.751148 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d80a0cf_2e96_4d22_b212_10ef2640d5e9.slice/crio-be687168183d33615f668e1063df2a6e990c88d3beaefc0f7b25bc60b208e3e8 WatchSource:0}: Error finding container be687168183d33615f668e1063df2a6e990c88d3beaefc0f7b25bc60b208e3e8: Status 404 returned error can't find the container with id be687168183d33615f668e1063df2a6e990c88d3beaefc0f7b25bc60b208e3e8 Oct 31 00:26:17.359619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.356341 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" event={"ID":"5d80a0cf-2e96-4d22-b212-10ef2640d5e9","Type":"ContainerStarted","Data":"6f0b5728a0137083d1f0bcb2b98890d02e547757a3714761b835109cfa12fa3d"} Oct 31 00:26:17.359619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.356391 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" event={"ID":"5d80a0cf-2e96-4d22-b212-10ef2640d5e9","Type":"ContainerStarted","Data":"be687168183d33615f668e1063df2a6e990c88d3beaefc0f7b25bc60b208e3e8"} Oct 31 00:26:17.444677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.444145 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9"] Oct 31 00:26:17.474634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.471097 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.517617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.516737 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" podStartSLOduration=1.5167176260000002 podStartE2EDuration="1.516717626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:17.515733502 +0000 UTC m=+1409.991091928" watchObservedRunningTime="2026-10-31 00:26:17.516717626 +0000 UTC m=+1409.992076051" Oct 31 00:26:17.686619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.677988 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.686619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.678077 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.686619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.678129 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.686619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.678156 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.686619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.678181 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db242-19e2-4882-a8f5-2375cc5a34c5-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.686619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.678212 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.686619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.678263 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.686619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.678378 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z269l\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-kube-api-access-z269l\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.686619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.678427 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.868696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.858135 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.868696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.858223 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.868696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.858280 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.868696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.858307 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.868696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.858337 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db242-19e2-4882-a8f5-2375cc5a34c5-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.868696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.858366 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.868696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.858419 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.868696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.858539 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z269l\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-kube-api-access-z269l\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.868696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.858618 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.885617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.882911 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.885617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.883554 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.943422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.886195 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.943422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.886879 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db242-19e2-4882-a8f5-2375cc5a34c5-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.943422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.887140 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.943422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.930222 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:17.998631 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:17.984744 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z269l\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-kube-api-access-z269l\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:18.062620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:18.057477 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:18.075617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:18.068154 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:18.111542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:18.082762 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f598e6e3-f02b-4750-b717-ff5ecba6477c" path="/var/lib/kubelet/pods/f598e6e3-f02b-4750-b717-ff5ecba6477c/volumes" Oct 31 00:26:18.119906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:18.119828 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9"] Oct 31 00:26:18.120407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:18.120278 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:18.420662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:18.420549 4969 generic.go:356] "Generic (PLEG): container finished" podID="5d80a0cf-2e96-4d22-b212-10ef2640d5e9" containerID="6f0b5728a0137083d1f0bcb2b98890d02e547757a3714761b835109cfa12fa3d" exitCode=0 Oct 31 00:26:18.420974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:18.420947 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" event={"ID":"5d80a0cf-2e96-4d22-b212-10ef2640d5e9","Type":"ContainerDied","Data":"6f0b5728a0137083d1f0bcb2b98890d02e547757a3714761b835109cfa12fa3d"} Oct 31 00:26:19.019861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.019793 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.068291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.068232 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 31 00:26:19.133098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.104249 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Oct 31 00:26:19.133827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.106342 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 31 00:26:19.261675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.257689 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded" Oct 31 00:26:19.289730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.288740 4969 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": context deadline exceeded" Oct 31 00:26:19.420623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.416790 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:19.673223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.673181 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:19.673753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.673727 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:19.673915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.673896 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:19.701037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:19.693868 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9"] Oct 31 00:26:20.056667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.041158 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:20.149763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.149720 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:20.163059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.163006 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 31 00:26:20.181508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.181480 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:26:20.181738 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.181720 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz"] Oct 31 00:26:20.181884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.181870 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:26:20.182040 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.182026 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:26:20.182163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.182150 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:26:20.182276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.182264 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce7e478-wz2lz"] Oct 31 00:26:20.609724 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.609685 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" event={"ID":"895db242-19e2-4882-a8f5-2375cc5a34c5","Type":"ContainerStarted","Data":"40242c1301ad25f8abdf37018dda4871a90b8a2e6b5373e5fc02fc484b3d6c4f"} Oct 31 00:26:20.683860 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.683544 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:26:20.694620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.694562 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:26:20.707757 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.707732 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:20.812564 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.812451 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-openshift-ca\") pod \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " Oct 31 00:26:20.820831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.820781 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-openshift-ca" pod "5d80a0cf-2e96-4d22-b212-10ef2640d5e9" (UID: "5d80a0cf-2e96-4d22-b212-10ef2640d5e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:20.821302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.821279 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-kube-api-access-c2hx4\" (UniqueName: \"kubernetes.io/projected/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-kube-api-access-c2hx4\") pod \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " Oct 31 00:26:20.822082 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.821555 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-xks-ca\") pod \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\" (UID: \"5d80a0cf-2e96-4d22-b212-10ef2640d5e9\") " Oct 31 00:26:20.825787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.825766 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:20.846522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.846489 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-xks-ca" pod "5d80a0cf-2e96-4d22-b212-10ef2640d5e9" (UID: "5d80a0cf-2e96-4d22-b212-10ef2640d5e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:20.847231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.847207 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-kube-api-access-c2hx4" pod "5d80a0cf-2e96-4d22-b212-10ef2640d5e9" (UID: "5d80a0cf-2e96-4d22-b212-10ef2640d5e9"). InnerVolumeSpecName "kube-api-access-c2hx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:20.942521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.939498 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2hx4\" (UniqueName: \"kubernetes.io/projected/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-kube-api-access-c2hx4\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:20.942521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:20.939543 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80a0cf-2e96-4d22-b212-10ef2640d5e9-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:21.620414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:21.620378 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" event={"ID":"895db242-19e2-4882-a8f5-2375cc5a34c5","Type":"ContainerStarted","Data":"f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1"} Oct 31 00:26:21.622011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:21.621989 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" event={"ID":"5d80a0cf-2e96-4d22-b212-10ef2640d5e9","Type":"ContainerDied","Data":"be687168183d33615f668e1063df2a6e990c88d3beaefc0f7b25bc60b208e3e8"} Oct 31 00:26:21.622113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:21.622022 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v" Oct 31 00:26:21.622113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:21.622029 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be687168183d33615f668e1063df2a6e990c88d3beaefc0f7b25bc60b208e3e8" Oct 31 00:26:21.734703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:21.734671 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c762c568-bcfa-4ec8-af87-c7fc1642075c" path="/var/lib/kubelet/pods/c762c568-bcfa-4ec8-af87-c7fc1642075c/volumes" Oct 31 00:26:21.743145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:21.743097 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" podStartSLOduration=4.743085305 podStartE2EDuration="4.743085305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:21.739082415 +0000 UTC m=+1414.214440838" watchObservedRunningTime="2026-10-31 00:26:21.743085305 +0000 UTC m=+1414.218443727" Oct 31 00:26:22.121313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.121281 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:22.122494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.122462 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:26:22.122685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.122526 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:26:22.853376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.853338 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2"] Oct 31 00:26:22.853952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.853938 4969 cpu_manager.go:414] "containerMap: removing container" podUID="5d80a0cf-2e96-4d22-b212-10ef2640d5e9" containerName="revoke-keys" Oct 31 00:26:22.853994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.853954 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d80a0cf-2e96-4d22-b212-10ef2640d5e9" containerName="revoke-keys" Oct 31 00:26:22.854045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.854040 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d80a0cf-2e96-4d22-b212-10ef2640d5e9" containerName="revoke-keys" Oct 31 00:26:22.858730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.858710 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:22.965853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.965813 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:22.966006 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.965887 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:22.966045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.965997 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:22.966045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.966029 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:22.966139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.966048 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79597fb-1903-4420-a542-1b036dd4e36c-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:22.966183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.966164 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmn5\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-kube-api-access-7hmn5\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:22.966230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.966218 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:22.966313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.966294 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79597fb-1903-4420-a542-1b036dd4e36c-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:22.966353 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:22.966341 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.016052 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.016017 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2"] Oct 31 00:26:23.067161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067127 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067197 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067222 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067248 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79597fb-1903-4420-a542-1b036dd4e36c-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067311 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmn5\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-kube-api-access-7hmn5\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067370 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067407 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79597fb-1903-4420-a542-1b036dd4e36c-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067443 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067522 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067646 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067725 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067938 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.067973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.067957 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.068298 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.068281 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79597fb-1903-4420-a542-1b036dd4e36c-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.069710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.069692 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.070142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.070123 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79597fb-1903-4420-a542-1b036dd4e36c-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.122668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.121613 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:26:23.122668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.121670 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:26:23.128288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.128261 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.128433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.128409 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmn5\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-kube-api-access-7hmn5\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.172950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.172918 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:23.637302 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:23.637269 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79597fb_1903_4420_a542_1b036dd4e36c.slice/crio-a2f3266b209f409e02f95f7d78eb76cd1638486ea520114fe591dc965b81547d WatchSource:0}: Error finding container a2f3266b209f409e02f95f7d78eb76cd1638486ea520114fe591dc965b81547d: Status 404 returned error can't find the container with id a2f3266b209f409e02f95f7d78eb76cd1638486ea520114fe591dc965b81547d Oct 31 00:26:23.637783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.637760 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2"] Oct 31 00:26:23.639775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.639743 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:23.639869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.639803 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:23.639869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:23.639833 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:24.121041 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:24.121010 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:26:24.121425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:24.121063 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:26:24.647269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:24.647228 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" event={"ID":"b79597fb-1903-4420-a542-1b036dd4e36c","Type":"ContainerStarted","Data":"b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc"} Oct 31 00:26:24.647521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:24.647501 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" event={"ID":"b79597fb-1903-4420-a542-1b036dd4e36c","Type":"ContainerStarted","Data":"a2f3266b209f409e02f95f7d78eb76cd1638486ea520114fe591dc965b81547d"} Oct 31 00:26:24.837781 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:24.834299 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" podStartSLOduration=2.834279313 podStartE2EDuration="2.834279313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:24.824107136 +0000 UTC m=+1417.299465560" watchObservedRunningTime="2026-10-31 00:26:24.834279313 +0000 UTC m=+1417.309637737" Oct 31 00:26:25.121185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:25.121140 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:26:25.121849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:25.121812 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:26:25.174093 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:25.174061 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:25.175941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:25.175904 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:26:25.176139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:25.176120 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 00:26:26.124618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:26.122001 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:26:26.124618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:26.122069 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:26:26.173786 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:26.173750 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:26:26.173947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:26.173796 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 00:26:27.122380 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:27.122262 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:26:27.122380 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:27.122344 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:26:27.174028 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:27.173994 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:26:27.174427 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:27.174060 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 00:26:28.174068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:28.174031 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:26:28.174620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:28.174098 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 00:26:28.556797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:28.556704 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:28.557698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:28.557671 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:28.558451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:28.558427 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:26:29.048120 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:29.048085 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f"] Oct 31 00:26:29.051979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:29.051954 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14116ff6-9f47f"] Oct 31 00:26:29.173997 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:29.173961 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:26:29.174167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:29.174023 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 00:26:29.734827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:29.734793 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ea92f5-9159-4a74-bfc2-275601c12cba" path="/var/lib/kubelet/pods/91ea92f5-9159-4a74-bfc2-275601c12cba/volumes" Oct 31 00:26:30.173613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:30.173571 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 00:26:30.173801 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:30.173668 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 00:26:31.000249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.000217 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q"] Oct 31 00:26:31.006003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.005977 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.037892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.037861 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4be4481-060e-4509-b829-5b765f345474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b2b1248-vfb7q\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.038154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.038122 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4be4481-060e-4509-b829-5b765f345474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b2b1248-vfb7q\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.038268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.038179 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxwl\" (UniqueName: \"kubernetes.io/projected/b4be4481-060e-4509-b829-5b765f345474-kube-api-access-qzxwl\") pod \"maas-api-revoke-sub-3b2b1248-vfb7q\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.064114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.064082 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q"] Oct 31 00:26:31.107832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.107792 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:31.107998 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.107891 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:26:31.139805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.139776 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4be4481-060e-4509-b829-5b765f345474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b2b1248-vfb7q\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.139957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.139818 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxwl\" (UniqueName: \"kubernetes.io/projected/b4be4481-060e-4509-b829-5b765f345474-kube-api-access-qzxwl\") pod \"maas-api-revoke-sub-3b2b1248-vfb7q\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.139957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.139880 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4be4481-060e-4509-b829-5b765f345474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b2b1248-vfb7q\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.140530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.140503 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4be4481-060e-4509-b829-5b765f345474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b2b1248-vfb7q\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.142355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.142333 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4be4481-060e-4509-b829-5b765f345474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b2b1248-vfb7q\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.178299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.178271 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:31.178741 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.178720 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:31.179326 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.179308 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:26:31.200669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.200645 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxwl\" (UniqueName: \"kubernetes.io/projected/b4be4481-060e-4509-b829-5b765f345474-kube-api-access-qzxwl\") pod \"maas-api-revoke-sub-3b2b1248-vfb7q\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.325753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.325681 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:31.715228 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:31.715193 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4be4481_060e_4509_b829_5b765f345474.slice/crio-e1e62b8808b15e9ecbc9cff1a371f0f752cf1eeb231e3af740f5e1dbdf66e713 WatchSource:0}: Error finding container e1e62b8808b15e9ecbc9cff1a371f0f752cf1eeb231e3af740f5e1dbdf66e713: Status 404 returned error can't find the container with id e1e62b8808b15e9ecbc9cff1a371f0f752cf1eeb231e3af740f5e1dbdf66e713 Oct 31 00:26:31.715906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:31.715878 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q"] Oct 31 00:26:32.607541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.607502 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9"] Oct 31 00:26:32.612035 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.612010 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.659812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.659789 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp"] Oct 31 00:26:32.660407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.660381 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.660491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.660423 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.660491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.660474 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgq7\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-kube-api-access-djgq7\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.661755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.660649 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.661755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.660796 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.661755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.660832 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32dde409-51ba-4784-9c04-3f9e8bf45f73-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.661755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.660954 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.661755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.661023 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.661755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.661091 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.664355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.664338 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.710213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.710166 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9"] Oct 31 00:26:32.734338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.734299 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp"] Oct 31 00:26:32.741564 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:32.741532 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:26:32.745032 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.745008 4969 generic.go:356] "Generic (PLEG): container finished" podID="b4be4481-060e-4509-b829-5b765f345474" containerID="8adf5e5ca13cc33e92ba3a6baec73f7414c9f1ae76deea277373f36fb1d6a96d" exitCode=0 Oct 31 00:26:32.745168 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.745102 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" event={"ID":"b4be4481-060e-4509-b829-5b765f345474","Type":"ContainerDied","Data":"8adf5e5ca13cc33e92ba3a6baec73f7414c9f1ae76deea277373f36fb1d6a96d"} Oct 31 00:26:32.745212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.745164 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" event={"ID":"b4be4481-060e-4509-b829-5b765f345474","Type":"ContainerStarted","Data":"e1e62b8808b15e9ecbc9cff1a371f0f752cf1eeb231e3af740f5e1dbdf66e713"} Oct 31 00:26:32.762864 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.762834 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.762974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.762884 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc817050-32a6-4b90-9a36-58c5be3da104-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.762974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.762919 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.762974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.762943 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc817050-32a6-4b90-9a36-58c5be3da104-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.763096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.762980 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.763096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763085 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.763167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763113 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.763167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763148 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgq7\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-kube-api-access-djgq7\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.763267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763180 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.763267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763211 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg62h\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-kube-api-access-bg62h\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.763267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763261 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.763410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763284 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.763470 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763405 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.763470 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763431 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.763571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763480 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.763571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763507 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32dde409-51ba-4784-9c04-3f9e8bf45f73-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.763571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763526 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.763571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763551 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.763804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763669 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.763804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.763735 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.764156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.764093 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.764156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.764118 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.764306 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.764172 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32dde409-51ba-4784-9c04-3f9e8bf45f73-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.765985 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.765951 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.766110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.766082 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.824242 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.824214 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.824373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.824348 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgq7\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-kube-api-access-djgq7\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.864937 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.864847 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc817050-32a6-4b90-9a36-58c5be3da104-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.864937 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.864908 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.864937 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.864933 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc817050-32a6-4b90-9a36-58c5be3da104-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.865223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.864977 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.865223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.865088 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.865223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.865124 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg62h\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-kube-api-access-bg62h\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.865223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.865182 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.865440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.865298 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.865440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.865337 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.865641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.865621 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc817050-32a6-4b90-9a36-58c5be3da104-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.865873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.865847 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.865974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.865950 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.866041 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.866022 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.866099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.866039 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.867440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.867415 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.867714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.867640 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc817050-32a6-4b90-9a36-58c5be3da104-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.925790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.925760 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.926027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.926005 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg62h\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-kube-api-access-bg62h\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:32.935958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.935928 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:32.977361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:32.977334 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:33.053712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.053678 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc"] Oct 31 00:26:33.057952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.057924 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da885d4-mx8tc"] Oct 31 00:26:33.333255 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.333222 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4"] Oct 31 00:26:33.338736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.338714 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.371946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.371918 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9"] Oct 31 00:26:33.372753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.372726 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e22c8b6-zmlh4\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.372853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.372766 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e22c8b6-zmlh4\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.372918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.372893 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95m58\" (UniqueName: \"kubernetes.io/projected/2542c347-1029-47fe-9acb-ded6f13945b2-kube-api-access-95m58\") pod \"maas-api-revoke-sub-2e22c8b6-zmlh4\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.373727 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:33.373699 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32dde409_51ba_4784_9c04_3f9e8bf45f73.slice/crio-452b82c9b2d001cc2de210ea0c73bb00bef2f3cbabe353cb190c860db81f5a30 WatchSource:0}: Error finding container 452b82c9b2d001cc2de210ea0c73bb00bef2f3cbabe353cb190c860db81f5a30: Status 404 returned error can't find the container with id 452b82c9b2d001cc2de210ea0c73bb00bef2f3cbabe353cb190c860db81f5a30 Oct 31 00:26:33.376304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.376259 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:33.376394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.376333 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:33.376394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.376374 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:33.395274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.395249 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4"] Oct 31 00:26:33.425664 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:33.425584 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc817050_32a6_4b90_9a36_58c5be3da104.slice/crio-2d71ae8e1bd4b557e02cc4f91c6a815f7b983c1f85081f96829f22572770578c WatchSource:0}: Error finding container 2d71ae8e1bd4b557e02cc4f91c6a815f7b983c1f85081f96829f22572770578c: Status 404 returned error can't find the container with id 2d71ae8e1bd4b557e02cc4f91c6a815f7b983c1f85081f96829f22572770578c Oct 31 00:26:33.425803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.425671 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp"] Oct 31 00:26:33.428269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.428238 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:33.428359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.428297 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:33.428359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.428325 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:26:33.474231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.474206 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e22c8b6-zmlh4\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.474338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.474237 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e22c8b6-zmlh4\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.474338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.474266 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95m58\" (UniqueName: \"kubernetes.io/projected/2542c347-1029-47fe-9acb-ded6f13945b2-kube-api-access-95m58\") pod \"maas-api-revoke-sub-2e22c8b6-zmlh4\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.475010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.474969 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e22c8b6-zmlh4\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.477178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.477155 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e22c8b6-zmlh4\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.536491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.536453 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95m58\" (UniqueName: \"kubernetes.io/projected/2542c347-1029-47fe-9acb-ded6f13945b2-kube-api-access-95m58\") pod \"maas-api-revoke-sub-2e22c8b6-zmlh4\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.654542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.654507 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:33.741636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.739249 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8bd090-269d-4be1-a34a-e975efac3d0a" path="/var/lib/kubelet/pods/9e8bd090-269d-4be1-a34a-e975efac3d0a/volumes" Oct 31 00:26:33.755739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.755645 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" event={"ID":"dc817050-32a6-4b90-9a36-58c5be3da104","Type":"ContainerStarted","Data":"8c70bf0ce2b9a1426fdb8a1f14262574890e97f27183aa56c33e33d165524d0a"} Oct 31 00:26:33.755739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.755707 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" event={"ID":"dc817050-32a6-4b90-9a36-58c5be3da104","Type":"ContainerStarted","Data":"2d71ae8e1bd4b557e02cc4f91c6a815f7b983c1f85081f96829f22572770578c"} Oct 31 00:26:33.757613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.757527 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" event={"ID":"32dde409-51ba-4784-9c04-3f9e8bf45f73","Type":"ContainerStarted","Data":"764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898"} Oct 31 00:26:33.757613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.757574 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" event={"ID":"32dde409-51ba-4784-9c04-3f9e8bf45f73","Type":"ContainerStarted","Data":"452b82c9b2d001cc2de210ea0c73bb00bef2f3cbabe353cb190c860db81f5a30"} Oct 31 00:26:33.878871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.878817 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podStartSLOduration=1.8788027029999999 podStartE2EDuration="1.878802703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:33.875817584 +0000 UTC m=+1426.351176007" watchObservedRunningTime="2026-10-31 00:26:33.878802703 +0000 UTC m=+1426.354161126" Oct 31 00:26:33.936412 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.936301 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:33.951993 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.951957 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:33.978766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:33.978730 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:34.000341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.000275 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podStartSLOduration=2.000252824 podStartE2EDuration="2.000252824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:33.996089619 +0000 UTC m=+1426.471448042" watchObservedRunningTime="2026-10-31 00:26:34.000252824 +0000 UTC m=+1426.475611249" Oct 31 00:26:34.047493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.047464 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4"] Oct 31 00:26:34.047681 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:34.047657 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2542c347_1029_47fe_9acb_ded6f13945b2.slice/crio-c6c2292d8a42bfea81cde472d63e89f4dc606137ebdae563150f550b6534e73d WatchSource:0}: Error finding container c6c2292d8a42bfea81cde472d63e89f4dc606137ebdae563150f550b6534e73d: Status 404 returned error can't find the container with id c6c2292d8a42bfea81cde472d63e89f4dc606137ebdae563150f550b6534e73d Oct 31 00:26:34.116618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.104516 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4be4481-060e-4509-b829-5b765f345474-kube-api-access-qzxwl\" (UniqueName: \"kubernetes.io/projected/b4be4481-060e-4509-b829-5b765f345474-kube-api-access-qzxwl\") pod \"b4be4481-060e-4509-b829-5b765f345474\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " Oct 31 00:26:34.116618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.104582 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4be4481-060e-4509-b829-5b765f345474-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4be4481-060e-4509-b829-5b765f345474-maas-api-openshift-ca\") pod \"b4be4481-060e-4509-b829-5b765f345474\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " Oct 31 00:26:34.116618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.104701 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4be4481-060e-4509-b829-5b765f345474-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4be4481-060e-4509-b829-5b765f345474-maas-api-xks-ca\") pod \"b4be4481-060e-4509-b829-5b765f345474\" (UID: \"b4be4481-060e-4509-b829-5b765f345474\") " Oct 31 00:26:34.129883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.129845 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4be4481-060e-4509-b829-5b765f345474-maas-api-openshift-ca" pod "b4be4481-060e-4509-b829-5b765f345474" (UID: "b4be4481-060e-4509-b829-5b765f345474"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:34.137827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.137792 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4be4481-060e-4509-b829-5b765f345474-kube-api-access-qzxwl" pod "b4be4481-060e-4509-b829-5b765f345474" (UID: "b4be4481-060e-4509-b829-5b765f345474"). InnerVolumeSpecName "kube-api-access-qzxwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:34.138649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.138585 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4be4481-060e-4509-b829-5b765f345474-maas-api-xks-ca" pod "b4be4481-060e-4509-b829-5b765f345474" (UID: "b4be4481-060e-4509-b829-5b765f345474"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:34.219679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.215642 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzxwl\" (UniqueName: \"kubernetes.io/projected/b4be4481-060e-4509-b829-5b765f345474-kube-api-access-qzxwl\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:34.220014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.215687 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4be4481-060e-4509-b829-5b765f345474-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:34.220014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.219985 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4be4481-060e-4509-b829-5b765f345474-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:34.766959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.766919 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" event={"ID":"b4be4481-060e-4509-b829-5b765f345474","Type":"ContainerDied","Data":"e1e62b8808b15e9ecbc9cff1a371f0f752cf1eeb231e3af740f5e1dbdf66e713"} Oct 31 00:26:34.767436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.766977 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1e62b8808b15e9ecbc9cff1a371f0f752cf1eeb231e3af740f5e1dbdf66e713" Oct 31 00:26:34.767436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.766933 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q" Oct 31 00:26:34.769200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.769171 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" event={"ID":"2542c347-1029-47fe-9acb-ded6f13945b2","Type":"ContainerStarted","Data":"110d6d702c2354ceffc2086bce3758b716b58e0b2445f6905fa639ff48718650"} Oct 31 00:26:34.769355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.769220 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" event={"ID":"2542c347-1029-47fe-9acb-ded6f13945b2","Type":"ContainerStarted","Data":"c6c2292d8a42bfea81cde472d63e89f4dc606137ebdae563150f550b6534e73d"} Oct 31 00:26:34.915676 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.914628 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" podStartSLOduration=1.910573525 podStartE2EDuration="1.910573525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:34.901904829 +0000 UTC m=+1427.377263254" watchObservedRunningTime="2026-10-31 00:26:34.910573525 +0000 UTC m=+1427.385931950" Oct 31 00:26:34.939887 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.939852 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:26:34.940051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:34.939910 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:26:35.015619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:35.005829 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:26:35.015619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:35.005887 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 00:26:35.821706 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:35.821670 4969 generic.go:356] "Generic (PLEG): container finished" podID="2542c347-1029-47fe-9acb-ded6f13945b2" containerID="110d6d702c2354ceffc2086bce3758b716b58e0b2445f6905fa639ff48718650" exitCode=0 Oct 31 00:26:35.823643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:35.823612 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" event={"ID":"2542c347-1029-47fe-9acb-ded6f13945b2","Type":"ContainerDied","Data":"110d6d702c2354ceffc2086bce3758b716b58e0b2445f6905fa639ff48718650"} Oct 31 00:26:35.936832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:35.936798 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:26:35.936994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:35.936850 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:26:35.979967 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:35.979936 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:26:35.980230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:35.980201 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 00:26:36.938644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:36.937367 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:26:36.938644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:36.937442 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:26:37.045776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.007242 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:26:37.045776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.007306 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 00:26:37.129885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.129854 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:37.268926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.268530 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-openshift-ca\") pod \"2542c347-1029-47fe-9acb-ded6f13945b2\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " Oct 31 00:26:37.268926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.268722 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-xks-ca\") pod \"2542c347-1029-47fe-9acb-ded6f13945b2\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " Oct 31 00:26:37.268926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.268849 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2542c347-1029-47fe-9acb-ded6f13945b2-kube-api-access-95m58\" (UniqueName: \"kubernetes.io/projected/2542c347-1029-47fe-9acb-ded6f13945b2-kube-api-access-95m58\") pod \"2542c347-1029-47fe-9acb-ded6f13945b2\" (UID: \"2542c347-1029-47fe-9acb-ded6f13945b2\") " Oct 31 00:26:37.268926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.268862 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-openshift-ca" pod "2542c347-1029-47fe-9acb-ded6f13945b2" (UID: "2542c347-1029-47fe-9acb-ded6f13945b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:37.269568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.269539 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:37.271429 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.271287 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-xks-ca" pod "2542c347-1029-47fe-9acb-ded6f13945b2" (UID: "2542c347-1029-47fe-9acb-ded6f13945b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:37.272238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.272218 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2542c347-1029-47fe-9acb-ded6f13945b2-kube-api-access-95m58" pod "2542c347-1029-47fe-9acb-ded6f13945b2" (UID: "2542c347-1029-47fe-9acb-ded6f13945b2"). InnerVolumeSpecName "kube-api-access-95m58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:37.370617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.370565 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2542c347-1029-47fe-9acb-ded6f13945b2-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:37.370617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.370605 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95m58\" (UniqueName: \"kubernetes.io/projected/2542c347-1029-47fe-9acb-ded6f13945b2-kube-api-access-95m58\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:37.849440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.849402 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" event={"ID":"2542c347-1029-47fe-9acb-ded6f13945b2","Type":"ContainerDied","Data":"c6c2292d8a42bfea81cde472d63e89f4dc606137ebdae563150f550b6534e73d"} Oct 31 00:26:37.849625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.849458 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6c2292d8a42bfea81cde472d63e89f4dc606137ebdae563150f550b6534e73d" Oct 31 00:26:37.849625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.849521 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4" Oct 31 00:26:37.937808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.937763 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:26:37.937992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:37.937813 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:26:38.032854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:38.032749 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:26:38.032854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:38.032816 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 00:26:38.936341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:38.936309 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:26:38.936493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:38.936365 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:26:38.985628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:38.984979 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:26:38.985628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:38.985038 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 00:26:39.936508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:39.936477 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:26:39.936906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:39.936531 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:26:39.978154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:39.978123 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:26:39.978327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:39.978178 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 00:26:40.941168 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:40.941129 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:40.941547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:40.941409 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:40.942121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:40.942103 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:26:40.982193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:40.982163 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:40.982479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:40.982462 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:40.983070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:40.983054 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:26:41.768764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.768728 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs"] Oct 31 00:26:41.771543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.769813 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b4be4481-060e-4509-b829-5b765f345474" containerName="revoke-keys" Oct 31 00:26:41.771707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.771692 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4be4481-060e-4509-b829-5b765f345474" containerName="revoke-keys" Oct 31 00:26:41.771870 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.771860 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2542c347-1029-47fe-9acb-ded6f13945b2" containerName="revoke-keys" Oct 31 00:26:41.771961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.771952 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2542c347-1029-47fe-9acb-ded6f13945b2" containerName="revoke-keys" Oct 31 00:26:41.772219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.772209 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4be4481-060e-4509-b829-5b765f345474" containerName="revoke-keys" Oct 31 00:26:41.772305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.772296 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2542c347-1029-47fe-9acb-ded6f13945b2" containerName="revoke-keys" Oct 31 00:26:41.776895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.776869 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:41.832444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.832414 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:41.834742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.834703 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:26:41.870245 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:41.870215 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs"] Oct 31 00:26:42.051667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.048038 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94d7f41a-6bfcs\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:42.051667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.048327 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf489\" (UniqueName: \"kubernetes.io/projected/fc571969-762d-4fad-8bd5-0cf8bbdf0755-kube-api-access-kf489\") pod \"maas-api-revoke-sub-94d7f41a-6bfcs\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:42.051667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.048436 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94d7f41a-6bfcs\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:42.152477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.152435 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf489\" (UniqueName: \"kubernetes.io/projected/fc571969-762d-4fad-8bd5-0cf8bbdf0755-kube-api-access-kf489\") pod \"maas-api-revoke-sub-94d7f41a-6bfcs\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:42.152835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.152808 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94d7f41a-6bfcs\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:42.153078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.153055 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94d7f41a-6bfcs\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:42.157661 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.157635 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94d7f41a-6bfcs\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:42.226451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.226382 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf489\" (UniqueName: \"kubernetes.io/projected/fc571969-762d-4fad-8bd5-0cf8bbdf0755-kube-api-access-kf489\") pod \"maas-api-revoke-sub-94d7f41a-6bfcs\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:42.229142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.229115 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94d7f41a-6bfcs\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:42.430620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:42.429885 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:43.060735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:43.060685 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:43.410760 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:43.410715 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 31 00:26:43.461718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:43.461585 4969 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:43.786308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:43.746437 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs"] Oct 31 00:26:43.786308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:43.747170 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 31 00:26:43.786308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:43.747521 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Oct 31 00:26:43.997484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:43.996988 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podUID="f8736325-6dcb-4b95-ac2c-a9a25635af0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 31 00:26:44.029013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.028983 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" event={"ID":"fc571969-762d-4fad-8bd5-0cf8bbdf0755","Type":"ContainerStarted","Data":"f176a54e20105e960f156e8a8e59c9f268cb7fb41287009d86b7f6e01cd06c68"} Oct 31 00:26:44.055084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.055056 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm"] Oct 31 00:26:44.059648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.059628 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b8fb89f-c7mtm"] Oct 31 00:26:44.287992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.287911 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c"] Oct 31 00:26:44.293249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.293233 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.347602 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.347562 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"kube-root-ca.crt\"" Oct 31 00:26:44.348246 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.348208 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"openshift-service-ca.crt\"" Oct 31 00:26:44.348392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.348209 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"default-dockercfg-zhdv9\"" Oct 31 00:26:44.348392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.348219 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7abb95\"/\"distinct-w3-7abb95-kserve-self-signed-certs\"" Oct 31 00:26:44.353040 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.353015 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c"] Oct 31 00:26:44.410521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.410495 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tmp-dir\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.410679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.410657 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-dshm\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.410748 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.410704 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-home\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.411119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.411093 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-model-cache\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.411218 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.411144 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tls-certs\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.411530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.411245 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxs5x\" (UniqueName: \"kubernetes.io/projected/ff3b5687-3b27-4a37-a867-73a2d0bec08a-kube-api-access-fxs5x\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.512989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.512956 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-model-cache\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.513158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.513000 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tls-certs\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.513158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.513032 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxs5x\" (UniqueName: \"kubernetes.io/projected/ff3b5687-3b27-4a37-a867-73a2d0bec08a-kube-api-access-fxs5x\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.513158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.513129 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tmp-dir\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.513334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.513193 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-dshm\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.513334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.513220 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-home\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.513460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.513435 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-model-cache\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.513541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.513521 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tmp-dir\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.513687 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.513669 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-home\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.515543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.515518 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-dshm\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.515772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.515755 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tls-certs\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.573934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.573871 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxs5x\" (UniqueName: \"kubernetes.io/projected/ff3b5687-3b27-4a37-a867-73a2d0bec08a-kube-api-access-fxs5x\") pod \"distinct-w3-7abb95-kserve-589b4c6b8d-8q45c\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.605273 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.605249 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:44.988282 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:44.988249 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c"] Oct 31 00:26:44.989118 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:44.989089 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff3b5687_3b27_4a37_a867_73a2d0bec08a.slice/crio-1e32a94422a98c82e0a9615164a4d92a223d80ab8463be24ed061a55acc48eaf WatchSource:0}: Error finding container 1e32a94422a98c82e0a9615164a4d92a223d80ab8463be24ed061a55acc48eaf: Status 404 returned error can't find the container with id 1e32a94422a98c82e0a9615164a4d92a223d80ab8463be24ed061a55acc48eaf Oct 31 00:26:45.038312 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:45.038280 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" event={"ID":"ff3b5687-3b27-4a37-a867-73a2d0bec08a","Type":"ContainerStarted","Data":"1e32a94422a98c82e0a9615164a4d92a223d80ab8463be24ed061a55acc48eaf"} Oct 31 00:26:45.040028 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:45.040004 4969 generic.go:356] "Generic (PLEG): container finished" podID="fc571969-762d-4fad-8bd5-0cf8bbdf0755" containerID="040ada1cd33cbeb14320ae4b92e6fe65471fb0b092b1b996b6e6881f7a6a1054" exitCode=0 Oct 31 00:26:45.040130 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:45.040072 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" event={"ID":"fc571969-762d-4fad-8bd5-0cf8bbdf0755","Type":"ContainerDied","Data":"040ada1cd33cbeb14320ae4b92e6fe65471fb0b092b1b996b6e6881f7a6a1054"} Oct 31 00:26:45.737424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:45.737390 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e6cd26-86fe-4089-ba09-144095cf16fb" path="/var/lib/kubelet/pods/03e6cd26-86fe-4089-ba09-144095cf16fb/volumes" Oct 31 00:26:46.058436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.058317 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" event={"ID":"ff3b5687-3b27-4a37-a867-73a2d0bec08a","Type":"ContainerStarted","Data":"143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54"} Oct 31 00:26:46.176666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.176616 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podStartSLOduration=2.176580072 podStartE2EDuration="2.176580072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:46.174371692 +0000 UTC m=+1438.649730125" watchObservedRunningTime="2026-10-31 00:26:46.176580072 +0000 UTC m=+1438.651938498" Oct 31 00:26:46.211517 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.211493 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:46.256841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.256812 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-openshift-ca\") pod \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " Oct 31 00:26:46.257020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.256991 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-xks-ca\") pod \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " Oct 31 00:26:46.257099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.257050 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc571969-762d-4fad-8bd5-0cf8bbdf0755-kube-api-access-kf489\" (UniqueName: \"kubernetes.io/projected/fc571969-762d-4fad-8bd5-0cf8bbdf0755-kube-api-access-kf489\") pod \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\" (UID: \"fc571969-762d-4fad-8bd5-0cf8bbdf0755\") " Oct 31 00:26:46.257191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.257168 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-openshift-ca" pod "fc571969-762d-4fad-8bd5-0cf8bbdf0755" (UID: "fc571969-762d-4fad-8bd5-0cf8bbdf0755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:46.257484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.257454 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.259488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.259464 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-xks-ca" pod "fc571969-762d-4fad-8bd5-0cf8bbdf0755" (UID: "fc571969-762d-4fad-8bd5-0cf8bbdf0755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:46.259573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.259532 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc571969-762d-4fad-8bd5-0cf8bbdf0755-kube-api-access-kf489" pod "fc571969-762d-4fad-8bd5-0cf8bbdf0755" (UID: "fc571969-762d-4fad-8bd5-0cf8bbdf0755"). InnerVolumeSpecName "kube-api-access-kf489". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.359161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.359090 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc571969-762d-4fad-8bd5-0cf8bbdf0755-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.359161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.359120 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf489\" (UniqueName: \"kubernetes.io/projected/fc571969-762d-4fad-8bd5-0cf8bbdf0755-kube-api-access-kf489\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.605891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:46.605864 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:47.068099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:47.068060 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" Oct 31 00:26:47.068099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:47.068087 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs" event={"ID":"fc571969-762d-4fad-8bd5-0cf8bbdf0755","Type":"ContainerDied","Data":"f176a54e20105e960f156e8a8e59c9f268cb7fb41287009d86b7f6e01cd06c68"} Oct 31 00:26:47.068632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:47.068142 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f176a54e20105e960f156e8a8e59c9f268cb7fb41287009d86b7f6e01cd06c68" Oct 31 00:26:48.106641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:48.106609 4969 scope.go:122] "RemoveContainer" containerID="e3f127b60a1fca17f3c9b07ab7b195c7a9fc978b699fd43716f9f519a19aadf2" Oct 31 00:26:48.117249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:48.117228 4969 scope.go:122] "RemoveContainer" containerID="178537bebab7b504160cecfbf05f87083f33ce271f0457e6857a7f9c8d4a3668" Oct 31 00:26:52.519554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.519517 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz"] Oct 31 00:26:52.520102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.520089 4969 cpu_manager.go:414] "containerMap: removing container" podUID="fc571969-762d-4fad-8bd5-0cf8bbdf0755" containerName="revoke-keys" Oct 31 00:26:52.520148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.520103 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc571969-762d-4fad-8bd5-0cf8bbdf0755" containerName="revoke-keys" Oct 31 00:26:52.520233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.520224 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc571969-762d-4fad-8bd5-0cf8bbdf0755" containerName="revoke-keys" Oct 31 00:26:52.525427 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.525406 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.584636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.584607 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz"] Oct 31 00:26:52.628792 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.628760 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:26:52.628968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.628807 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:52.636779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.636752 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc036826-6054-4c9d-92f7-40500e541871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0010c9c1-rlnsz\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.636866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.636816 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc036826-6054-4c9d-92f7-40500e541871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0010c9c1-rlnsz\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.636974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.636952 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hxq\" (UniqueName: \"kubernetes.io/projected/fc036826-6054-4c9d-92f7-40500e541871-kube-api-access-t2hxq\") pod \"maas-api-revoke-sub-0010c9c1-rlnsz\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.738789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.738759 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc036826-6054-4c9d-92f7-40500e541871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0010c9c1-rlnsz\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.738972 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.738813 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc036826-6054-4c9d-92f7-40500e541871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0010c9c1-rlnsz\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.738972 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.738873 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hxq\" (UniqueName: \"kubernetes.io/projected/fc036826-6054-4c9d-92f7-40500e541871-kube-api-access-t2hxq\") pod \"maas-api-revoke-sub-0010c9c1-rlnsz\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.739368 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.739345 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc036826-6054-4c9d-92f7-40500e541871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0010c9c1-rlnsz\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.741340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.741318 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc036826-6054-4c9d-92f7-40500e541871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0010c9c1-rlnsz\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.803128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.803067 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hxq\" (UniqueName: \"kubernetes.io/projected/fc036826-6054-4c9d-92f7-40500e541871-kube-api-access-t2hxq\") pod \"maas-api-revoke-sub-0010c9c1-rlnsz\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:52.839231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:52.839204 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:53.223377 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:26:53.223346 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc036826_6054_4c9d_92f7_40500e541871.slice/crio-ba31aa9ab3313cdf80bdf08a9d0847d26914cb45ac1f471418ca37a55a0eaa7b WatchSource:0}: Error finding container ba31aa9ab3313cdf80bdf08a9d0847d26914cb45ac1f471418ca37a55a0eaa7b: Status 404 returned error can't find the container with id ba31aa9ab3313cdf80bdf08a9d0847d26914cb45ac1f471418ca37a55a0eaa7b Oct 31 00:26:53.224056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:53.224036 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz"] Oct 31 00:26:54.086509 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:26:54.086479 4969 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc036826_6054_4c9d_92f7_40500e541871.slice/crio-a935867082154c5cdbb6baa4c21e9035b64d2c3172fe1b4b9775191a158c56cc.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:54.123898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:54.123867 4969 generic.go:356] "Generic (PLEG): container finished" podID="fc036826-6054-4c9d-92f7-40500e541871" containerID="a935867082154c5cdbb6baa4c21e9035b64d2c3172fe1b4b9775191a158c56cc" exitCode=0 Oct 31 00:26:54.124059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:54.123942 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" event={"ID":"fc036826-6054-4c9d-92f7-40500e541871","Type":"ContainerDied","Data":"a935867082154c5cdbb6baa4c21e9035b64d2c3172fe1b4b9775191a158c56cc"} Oct 31 00:26:54.124059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:54.124015 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" event={"ID":"fc036826-6054-4c9d-92f7-40500e541871","Type":"ContainerStarted","Data":"ba31aa9ab3313cdf80bdf08a9d0847d26914cb45ac1f471418ca37a55a0eaa7b"} Oct 31 00:26:54.606152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:54.606120 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:54.618953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:54.618927 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:55.143748 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.143721 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:26:55.295182 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.295152 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:55.371903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.371880 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc036826-6054-4c9d-92f7-40500e541871-kube-api-access-t2hxq\" (UniqueName: \"kubernetes.io/projected/fc036826-6054-4c9d-92f7-40500e541871-kube-api-access-t2hxq\") pod \"fc036826-6054-4c9d-92f7-40500e541871\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " Oct 31 00:26:55.372061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.371915 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc036826-6054-4c9d-92f7-40500e541871-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc036826-6054-4c9d-92f7-40500e541871-maas-api-openshift-ca\") pod \"fc036826-6054-4c9d-92f7-40500e541871\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " Oct 31 00:26:55.372061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.371972 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc036826-6054-4c9d-92f7-40500e541871-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc036826-6054-4c9d-92f7-40500e541871-maas-api-xks-ca\") pod \"fc036826-6054-4c9d-92f7-40500e541871\" (UID: \"fc036826-6054-4c9d-92f7-40500e541871\") " Oct 31 00:26:55.372284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.372263 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc036826-6054-4c9d-92f7-40500e541871-maas-api-openshift-ca" pod "fc036826-6054-4c9d-92f7-40500e541871" (UID: "fc036826-6054-4c9d-92f7-40500e541871"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:55.372486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.372472 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc036826-6054-4c9d-92f7-40500e541871-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:55.374193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.374168 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc036826-6054-4c9d-92f7-40500e541871-maas-api-xks-ca" pod "fc036826-6054-4c9d-92f7-40500e541871" (UID: "fc036826-6054-4c9d-92f7-40500e541871"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:55.374289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.374208 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc036826-6054-4c9d-92f7-40500e541871-kube-api-access-t2hxq" pod "fc036826-6054-4c9d-92f7-40500e541871" (UID: "fc036826-6054-4c9d-92f7-40500e541871"). InnerVolumeSpecName "kube-api-access-t2hxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:55.473438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.473405 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2hxq\" (UniqueName: \"kubernetes.io/projected/fc036826-6054-4c9d-92f7-40500e541871-kube-api-access-t2hxq\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:55.473438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:55.473433 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc036826-6054-4c9d-92f7-40500e541871-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:26:56.142126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:56.142093 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" Oct 31 00:26:56.142289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:56.142122 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz" event={"ID":"fc036826-6054-4c9d-92f7-40500e541871","Type":"ContainerDied","Data":"ba31aa9ab3313cdf80bdf08a9d0847d26914cb45ac1f471418ca37a55a0eaa7b"} Oct 31 00:26:56.142289 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:56.142165 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba31aa9ab3313cdf80bdf08a9d0847d26914cb45ac1f471418ca37a55a0eaa7b" Oct 31 00:26:57.501582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:57.494746 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd"] Oct 31 00:26:57.501582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:57.495684 4969 cpu_manager.go:414] "containerMap: removing container" podUID="fc036826-6054-4c9d-92f7-40500e541871" containerName="revoke-keys" Oct 31 00:26:57.501582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:57.495704 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc036826-6054-4c9d-92f7-40500e541871" containerName="revoke-keys" Oct 31 00:26:57.501582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:57.495849 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc036826-6054-4c9d-92f7-40500e541871" containerName="revoke-keys" Oct 31 00:26:57.624270 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:57.624227 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:57.729696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:57.724099 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f7c92a85-openshift-default-dockercfg-rt7pg\"" Oct 31 00:26:58.045619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.044984 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.045619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.045086 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.045619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.045152 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.045619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.045218 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b7814a-5fef-47e6-83a9-b1c511303a04-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.045619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.045273 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.045619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.045327 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.045619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.045383 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkzd\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-kube-api-access-xvkzd\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.045619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.045450 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.045619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.045510 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.397619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.379383 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.403760 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.403724 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.415862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.415824 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.419739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.419706 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.450623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.425690 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.450979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.450956 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b7814a-5fef-47e6-83a9-b1c511303a04-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.451148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.451132 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.451286 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.451272 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.451409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.451394 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkzd\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-kube-api-access-xvkzd\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.451921 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.391103 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.467632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.454651 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.467632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.455163 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.467632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.420483 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd"] Oct 31 00:26:58.467632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.455910 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b7814a-5fef-47e6-83a9-b1c511303a04-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.467632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.450823 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 31 00:26:58.467632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.459166 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.538635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.513025 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.565613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.565568 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.607633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.607575 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkzd\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-kube-api-access-xvkzd\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.672665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.669321 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.695545 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.695506 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:58.701957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.701931 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:26:58.734326 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.725907 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Oct 31 00:26:58.809635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:58.806273 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:59.069812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.069707 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:59.088039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.088000 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podUID="f8736325-6dcb-4b95-ac2c-a9a25635af0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:59.431620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.426680 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:59.440831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.440789 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Oct 31 00:26:59.455771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.455732 4969 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:59.456126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.456086 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 31 00:26:59.456717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.456692 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:26:59.611884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.611746 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:26:59.611884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.611811 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:59.813679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.759062 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 31 00:26:59.854056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:26:59.853999 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Oct 31 00:27:00.092471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.092355 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 31 00:27:00.388445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.386036 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd"] Oct 31 00:27:00.388445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.386167 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:27:00.388445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.386214 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:27:00.388445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.386263 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:27:00.428222 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.428198 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:27:00.428827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.428796 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:00.429277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.429257 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:27:00.461042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.461016 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:27:00.496875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.496826 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:00.497059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.496944 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:00.497059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.496995 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:00.611613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.607383 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:27:00.625810 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.625733 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:27:00.672846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.671734 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:27:00.873671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:00.870026 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" event={"ID":"a0b7814a-5fef-47e6-83a9-b1c511303a04","Type":"ContainerStarted","Data":"baf65187c089efb18d903f97b68e9c1556a120cb44057337b643be924769c6b9"} Oct 31 00:27:01.879163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:01.879121 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" event={"ID":"a0b7814a-5fef-47e6-83a9-b1c511303a04","Type":"ContainerStarted","Data":"fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21"} Oct 31 00:27:02.003561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.003507 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" podStartSLOduration=5.003492042 podStartE2EDuration="5.003492042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:02.001381751 +0000 UTC m=+1454.476740173" watchObservedRunningTime="2026-10-31 00:27:02.003492042 +0000 UTC m=+1454.478850465" Oct 31 00:27:02.727289 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:27:02.727253 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:27:02.809929 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.809897 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:27:02.815084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.815052 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw"] Oct 31 00:27:02.819586 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.819564 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:27:02.819726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.819653 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.894502 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.889573 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:27:02.894502 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.890646 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:27:02.923637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.916798 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.923637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.916946 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.923637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.917034 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.923637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.917160 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf045f-a7b5-4051-92a4-67956ad461a4-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.923637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.917221 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.923637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.917270 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.923637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.917296 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvh99\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-kube-api-access-vvh99\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.923637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.917451 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.923637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.917485 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:02.927336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:02.927313 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw"] Oct 31 00:27:03.018774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.018685 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.018774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.018731 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.018776 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.018826 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf045f-a7b5-4051-92a4-67956ad461a4-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.018862 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.018891 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.018907 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvh99\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-kube-api-access-vvh99\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.019022 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.019051 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.019148 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.019184 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.019207 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019513 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.019484 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.019845 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.019564 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf045f-a7b5-4051-92a4-67956ad461a4-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.021447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.021427 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.021530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.021514 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.083106 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.083062 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.083302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.083115 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvh99\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-kube-api-access-vvh99\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.133828 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.133796 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:03.580892 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:27:03.580849 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cf045f_a7b5_4051_92a4_67956ad461a4.slice/crio-9d3b386aa0e2c5ab598aa31b3305763ae55ce816b72bdfe56e3a655e5312cb9e WatchSource:0}: Error finding container 9d3b386aa0e2c5ab598aa31b3305763ae55ce816b72bdfe56e3a655e5312cb9e: Status 404 returned error can't find the container with id 9d3b386aa0e2c5ab598aa31b3305763ae55ce816b72bdfe56e3a655e5312cb9e Oct 31 00:27:03.582529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.582486 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw"] Oct 31 00:27:03.583383 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.583355 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:03.583439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.583417 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:03.583478 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.583447 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:03.908795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.908753 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" event={"ID":"64cf045f-a7b5-4051-92a4-67956ad461a4","Type":"ContainerStarted","Data":"18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c"} Oct 31 00:27:03.909190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:03.908855 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" event={"ID":"64cf045f-a7b5-4051-92a4-67956ad461a4","Type":"ContainerStarted","Data":"9d3b386aa0e2c5ab598aa31b3305763ae55ce816b72bdfe56e3a655e5312cb9e"} Oct 31 00:27:04.040004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:04.039955 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" podStartSLOduration=2.039939409 podStartE2EDuration="2.039939409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:04.034797373 +0000 UTC m=+1456.510155795" watchObservedRunningTime="2026-10-31 00:27:04.039939409 +0000 UTC m=+1456.515297831" Oct 31 00:27:04.134991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:04.134959 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:04.136271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:04.136240 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 00:27:04.136392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:04.136300 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 00:27:05.134842 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:05.134807 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 00:27:05.135323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:05.134871 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 00:27:06.134680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:06.134639 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 00:27:06.134881 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:06.134702 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 00:27:07.134453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.134422 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 00:27:07.134626 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.134473 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 00:27:07.868923 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.868874 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6"] Oct 31 00:27:07.873669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.873645 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:07.915635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.911656 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:07.915635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.911715 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:07.915635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.911794 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9h8\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-kube-api-access-5d9h8\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:07.915635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.911885 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:07.915635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.912020 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:07.915635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.912087 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:07.915635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.912124 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:07.915635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.912180 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:07.915635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:07.912222 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.011620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.008676 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6"] Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.013351 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.013441 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.013478 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.013500 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.013529 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.013552 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.013617 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.013634 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.013673 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9h8\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-kube-api-access-5d9h8\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.014244 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.014783 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.015053 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.015628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.015407 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.016343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.015833 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.019077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.017171 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.019077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.017240 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.075867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.075842 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.102248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.102209 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9h8\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-kube-api-access-5d9h8\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.134736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.134655 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 00:27:08.134736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.134712 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 00:27:08.221087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.221053 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:08.750816 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.727036 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6"] Oct 31 00:27:08.766529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.766129 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:08.766529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.766219 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:08.766529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.766265 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:08.958118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.958014 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" event={"ID":"fd8769f9-620c-470c-9f4a-c76f5262e9d9","Type":"ContainerStarted","Data":"4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8"} Oct 31 00:27:08.958118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:08.958084 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" event={"ID":"fd8769f9-620c-470c-9f4a-c76f5262e9d9","Type":"ContainerStarted","Data":"fa197312ef50aff59708d5145bf13d3fd9510a596883d17d5e8f2633f2672e9c"} Oct 31 00:27:09.134535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:09.134495 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 00:27:09.134721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:09.134561 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 00:27:09.233671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:09.233392 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:09.237212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:09.237180 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:27:09.242428 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:09.242393 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 00:27:10.134778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:10.134740 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 00:27:10.135276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:10.134817 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 00:27:10.222233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:10.222197 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:27:10.222539 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:10.222508 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 00:27:11.222114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:11.222078 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:27:11.222585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:11.222139 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 00:27:11.497817 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:11.497728 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:11.504676 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:11.504643 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:11.509805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:11.509787 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:27:11.719802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:11.719749 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" podStartSLOduration=4.719734828 podStartE2EDuration="4.719734828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:09.102578519 +0000 UTC m=+1461.577936944" watchObservedRunningTime="2026-10-31 00:27:11.719734828 +0000 UTC m=+1464.195093250" Oct 31 00:27:12.221874 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.221836 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:27:12.222050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.221903 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 00:27:12.325126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.325096 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd"] Oct 31 00:27:12.329300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.329281 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.406016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.405976 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:27:12.406016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.406010 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:27:12.413257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.413221 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd"] Oct 31 00:27:12.413742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.413715 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d786b006-c4jtd\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.413889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.413874 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d786b006-c4jtd\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.413984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.413972 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5fmm\" (UniqueName: \"kubernetes.io/projected/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-kube-api-access-v5fmm\") pod \"maas-api-revoke-sub-d786b006-c4jtd\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.518644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.518540 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5fmm\" (UniqueName: \"kubernetes.io/projected/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-kube-api-access-v5fmm\") pod \"maas-api-revoke-sub-d786b006-c4jtd\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.518803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.518721 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d786b006-c4jtd\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.518854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.518837 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d786b006-c4jtd\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.519697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.519674 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d786b006-c4jtd\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.522434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.522393 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d786b006-c4jtd\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.602354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.602318 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5fmm\" (UniqueName: \"kubernetes.io/projected/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-kube-api-access-v5fmm\") pod \"maas-api-revoke-sub-d786b006-c4jtd\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:12.643827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:12.643802 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:13.051552 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:27:13.051522 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339bf8b5_e3a2_4db5_b15a_5df2b2198bef.slice/crio-dfb58dcc55feb125bb68d7c043f5815f445de279252a76f0ae5836bda4de1b64 WatchSource:0}: Error finding container dfb58dcc55feb125bb68d7c043f5815f445de279252a76f0ae5836bda4de1b64: Status 404 returned error can't find the container with id dfb58dcc55feb125bb68d7c043f5815f445de279252a76f0ae5836bda4de1b64 Oct 31 00:27:13.052000 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:13.051977 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd"] Oct 31 00:27:13.221811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:13.221779 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:27:13.221979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:13.221832 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 00:27:14.032938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:14.032902 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" event={"ID":"339bf8b5-e3a2-4db5-b15a-5df2b2198bef","Type":"ContainerStarted","Data":"791ec20ac4b24f69d5e6edf42701e4ade253ad89e68d4e7e0dfc7de14700120e"} Oct 31 00:27:14.033438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:14.032967 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" event={"ID":"339bf8b5-e3a2-4db5-b15a-5df2b2198bef","Type":"ContainerStarted","Data":"dfb58dcc55feb125bb68d7c043f5815f445de279252a76f0ae5836bda4de1b64"} Oct 31 00:27:14.157767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:14.157695 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" podStartSLOduration=2.157677168 podStartE2EDuration="2.157677168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:14.153912123 +0000 UTC m=+1466.629270549" watchObservedRunningTime="2026-10-31 00:27:14.157677168 +0000 UTC m=+1466.633035599" Oct 31 00:27:14.222090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:14.222055 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:27:14.222265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:14.222119 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 00:27:15.042407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:15.042365 4969 generic.go:356] "Generic (PLEG): container finished" podID="339bf8b5-e3a2-4db5-b15a-5df2b2198bef" containerID="791ec20ac4b24f69d5e6edf42701e4ade253ad89e68d4e7e0dfc7de14700120e" exitCode=0 Oct 31 00:27:15.042889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:15.042451 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" event={"ID":"339bf8b5-e3a2-4db5-b15a-5df2b2198bef","Type":"ContainerDied","Data":"791ec20ac4b24f69d5e6edf42701e4ade253ad89e68d4e7e0dfc7de14700120e"} Oct 31 00:27:15.221453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:15.221417 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:27:15.221626 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:15.221502 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 00:27:15.969193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:15.969159 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb"] Oct 31 00:27:15.973649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:15.973624 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.065783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.065739 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.066236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.065875 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c719883-0029-4053-b988-c48f5248e09a-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.066236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.065995 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.066236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.066165 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgmkg\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-kube-api-access-kgmkg\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.066387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.066245 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.066387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.066366 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.066466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.066401 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.066571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.066548 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.066699 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.066680 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c719883-0029-4053-b988-c48f5248e09a-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.075646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.075584 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7abb95-openshift-default-dockercfg-d8tvz\"" Oct 31 00:27:16.084639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.084551 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28"] Oct 31 00:27:16.093239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.093170 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba3f881-74b28"] Oct 31 00:27:16.103163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.103134 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb"] Oct 31 00:27:16.168949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.168894 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.169250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.169230 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c719883-0029-4053-b988-c48f5248e09a-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.169494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.169304 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.169679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.169661 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.170699 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.170650 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c719883-0029-4053-b988-c48f5248e09a-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.170880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.169837 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c719883-0029-4053-b988-c48f5248e09a-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.171359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.171030 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.171586 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.171324 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.171859 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.171838 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmkg\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-kube-api-access-kgmkg\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.172399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.172377 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.172628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.172580 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.172790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.172773 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.172970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.172945 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.173121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.172691 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c719883-0029-4053-b988-c48f5248e09a-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.173500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.173368 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.173958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.173928 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.245008 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.244878 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.290743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.290710 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgmkg\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-kube-api-access-kgmkg\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.334836 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.334311 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:16.490978 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.490940 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-xks-ca\") pod \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " Oct 31 00:27:16.491650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.491626 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-kube-api-access-v5fmm\" (UniqueName: \"kubernetes.io/projected/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-kube-api-access-v5fmm\") pod \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " Oct 31 00:27:16.495349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.495274 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-openshift-ca\") pod \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\" (UID: \"339bf8b5-e3a2-4db5-b15a-5df2b2198bef\") " Oct 31 00:27:16.496292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.495868 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-openshift-ca" pod "339bf8b5-e3a2-4db5-b15a-5df2b2198bef" (UID: "339bf8b5-e3a2-4db5-b15a-5df2b2198bef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:16.497057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.497022 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:27:16.508633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.504996 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-xks-ca" pod "339bf8b5-e3a2-4db5-b15a-5df2b2198bef" (UID: "339bf8b5-e3a2-4db5-b15a-5df2b2198bef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:16.510462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.510405 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-kube-api-access-v5fmm" pod "339bf8b5-e3a2-4db5-b15a-5df2b2198bef" (UID: "339bf8b5-e3a2-4db5-b15a-5df2b2198bef"). InnerVolumeSpecName "kube-api-access-v5fmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:16.559672 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.555950 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:16.559672 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.557504 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:16.562234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.562167 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:27:16.589108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.589076 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:16.599201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.599169 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:27:16.599329 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:16.599207 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5fmm\" (UniqueName: \"kubernetes.io/projected/339bf8b5-e3a2-4db5-b15a-5df2b2198bef-kube-api-access-v5fmm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:27:17.035145 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:27:17.035107 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c719883_0029_4053_b988_c48f5248e09a.slice/crio-423b3c20e455d24ccbff7be0a7c1f0c7f476a77e698a362beb60906fbc26e7b2 WatchSource:0}: Error finding container 423b3c20e455d24ccbff7be0a7c1f0c7f476a77e698a362beb60906fbc26e7b2: Status 404 returned error can't find the container with id 423b3c20e455d24ccbff7be0a7c1f0c7f476a77e698a362beb60906fbc26e7b2 Oct 31 00:27:17.035934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:17.035909 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb"] Oct 31 00:27:17.037789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:17.037756 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:17.037854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:17.037824 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:17.037886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:17.037851 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:27:17.065219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:17.065197 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" Oct 31 00:27:17.065332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:17.065232 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd" event={"ID":"339bf8b5-e3a2-4db5-b15a-5df2b2198bef","Type":"ContainerDied","Data":"dfb58dcc55feb125bb68d7c043f5815f445de279252a76f0ae5836bda4de1b64"} Oct 31 00:27:17.065332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:17.065283 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfb58dcc55feb125bb68d7c043f5815f445de279252a76f0ae5836bda4de1b64" Oct 31 00:27:17.066607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:17.066571 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" event={"ID":"9c719883-0029-4053-b988-c48f5248e09a","Type":"ContainerStarted","Data":"423b3c20e455d24ccbff7be0a7c1f0c7f476a77e698a362beb60906fbc26e7b2"} Oct 31 00:27:17.737387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:17.737357 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96c9c7c-db60-4aef-928b-a0723a6ee8bc" path="/var/lib/kubelet/pods/f96c9c7c-db60-4aef-928b-a0723a6ee8bc/volumes" Oct 31 00:27:18.082674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:18.078689 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" event={"ID":"9c719883-0029-4053-b988-c48f5248e09a","Type":"ContainerStarted","Data":"beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8"} Oct 31 00:27:18.267532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:18.267476 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podStartSLOduration=3.267462745 podStartE2EDuration="3.267462745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:18.262309768 +0000 UTC m=+1470.737668190" watchObservedRunningTime="2026-10-31 00:27:18.267462745 +0000 UTC m=+1470.742821168" Oct 31 00:27:18.597613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:18.590990 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:18.601768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:18.601726 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:27:18.601916 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:18.601813 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 00:27:19.595621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:19.591934 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:27:19.595621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:19.592010 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 00:27:20.592618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:20.592126 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:27:20.592618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:20.592241 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 00:27:21.474619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:21.473712 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:21.525622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:21.524723 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:21.601620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:21.598863 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:27:21.601620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:21.598970 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 00:27:21.624618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:21.611686 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Oct 31 00:27:21.635784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:21.635737 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:21.692655 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:21.678701 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:21.776676 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:21.772667 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.000614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:21.991743 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podUID="f8736325-6dcb-4b95-ac2c-a9a25635af0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.240643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.216061 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded" Oct 31 00:27:22.482002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.481941 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.483903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.483654 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:27:22.528870 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.528775 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.529506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.529483 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:27:22.600568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.600533 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:27:22.600893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.600851 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 00:27:22.623879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.623838 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.634098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.634069 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:27:22.641711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.641667 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.642265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.642244 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:27:22.680720 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.680678 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.681344 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.681320 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:27:22.850678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.815177 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 31 00:27:22.938643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:22.898806 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:27:23.052616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.031696 4969 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.181642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.178650 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podUID="f8736325-6dcb-4b95-ac2c-a9a25635af0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.181642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.180173 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:27:23.230771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.230726 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.232015 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.231988 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:27:23.491090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.490886 4969 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded" Oct 31 00:27:23.531844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.531802 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": context deadline exceeded" Oct 31 00:27:23.535740 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.535705 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 31 00:27:23.634885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.634846 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Oct 31 00:27:23.650721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.650661 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.680276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.680241 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:27:23.680557 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.680519 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 00:27:23.700474 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.681276 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:23.945159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:23.927716 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.232918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.232810 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.234251 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.234205 4969 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.234532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.234507 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podUID="f8736325-6dcb-4b95-ac2c-a9a25635af0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.541840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.541693 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Oct 31 00:27:24.542974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.542940 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.591077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.591044 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:27:24.591351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.591308 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 00:27:24.638202 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.638151 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.687639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.680484 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded" Oct 31 00:27:24.713682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.708683 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.881624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.880902 4969 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podUID="f8736325-6dcb-4b95-ac2c-a9a25635af0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:24.977620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:24.949028 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 31 00:27:25.117887 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.117846 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6bnvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:27:25.118132 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.118103 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.268684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.253816 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.388638 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.336581 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podUID="f8736325-6dcb-4b95-ac2c-a9a25635af0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.551699 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.543680 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.551699 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.545359 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.601215 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.601071 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:27:25.601215 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.601132 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 00:27:25.656671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.633710 4969 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.656671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.655894 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.697671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.689689 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.725748 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.725704 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.846780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.846689 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-thkzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:27:25.850697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.850660 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:25.955617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:25.954672 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:26.173626 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.171361 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 00:27:26.210904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.210873 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 00:27:26.239520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.237332 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" Oct 31 00:27:26.269255 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.269214 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:26.287842 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.276387 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 00:27:26.287842 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.278843 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 00:27:26.296892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.296858 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 00:27:26.326789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.326748 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 00:27:26.419127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.419096 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 00:27:26.589847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.589758 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:27:26.589847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:26.589821 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 00:27:27.593556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:27.593522 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:27.594040 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:27.594011 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:27.594570 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:27.594549 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:27:32.727626 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:27:32.727567 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:27:42.033753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:42.033712 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:27:42.034359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:42.034322 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:42.036118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:42.036081 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:27:42.036362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:42.036336 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:42.273617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:42.266695 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podUID="f8736325-6dcb-4b95-ac2c-a9a25635af0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:42.311702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:42.287015 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:42.425220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:42.423811 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:42.571808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:42.571705 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:47.955298 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:47.955256 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:27:47.959523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:47.959498 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:27:47.973582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:47.973551 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:27:47.978009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:47.977990 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:27:48.161577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:48.161545 4969 scope.go:122] "RemoveContainer" containerID="da305d8bd754994efe3b05753db426e31eaae55167b0b628a4fee0f67a227e2a" Oct 31 00:27:48.172780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:48.172762 4969 scope.go:122] "RemoveContainer" containerID="8ffcffa3b96071cef1643774d0c7d16bd783f4f74e0e74796beae24f6903ea88" Oct 31 00:27:48.183333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:48.183313 4969 scope.go:122] "RemoveContainer" containerID="07fd95d2214ce754178f2ded975d1ff8dffc08c42bfa4c8c73ff1efe031700ba" Oct 31 00:27:48.193136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:27:48.193120 4969 scope.go:122] "RemoveContainer" containerID="3c000120b8a352c8a24aad0a2e45db9ebb5a42c78d1b5177cab21309c88e851e" Oct 31 00:28:02.732008 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:28:02.731970 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:28:09.576250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:09.576221 4969 generic.go:356] "Generic (PLEG): container finished" podID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerID="fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21" exitCode=0 Oct 31 00:28:09.576743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:09.576296 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" event={"ID":"a0b7814a-5fef-47e6-83a9-b1c511303a04","Type":"ContainerDied","Data":"fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21"} Oct 31 00:28:09.576903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:09.576873 4969 scope.go:122] "RemoveContainer" containerID="fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21" Oct 31 00:28:09.578438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:09.578414 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:28:09.578506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:09.578479 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:28:09.578545 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:09.578513 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:28:10.017704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.017669 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg"] Oct 31 00:28:10.018354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.018340 4969 cpu_manager.go:414] "containerMap: removing container" podUID="339bf8b5-e3a2-4db5-b15a-5df2b2198bef" containerName="revoke-keys" Oct 31 00:28:10.018405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.018356 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339bf8b5-e3a2-4db5-b15a-5df2b2198bef" containerName="revoke-keys" Oct 31 00:28:10.018486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.018477 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="339bf8b5-e3a2-4db5-b15a-5df2b2198bef" containerName="revoke-keys" Oct 31 00:28:10.021151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.021131 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.073677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.073649 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:28:10.074946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.074410 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:28:10.082970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.082937 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg"] Oct 31 00:28:10.110466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.110431 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69269ffe-3005-4849-abd1-97462a39286f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c916d419-mllrg\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.110678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.110490 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scm46\" (UniqueName: \"kubernetes.io/projected/69269ffe-3005-4849-abd1-97462a39286f-kube-api-access-scm46\") pod \"maas-api-revoke-sub-c916d419-mllrg\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.110678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.110561 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69269ffe-3005-4849-abd1-97462a39286f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c916d419-mllrg\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.212336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.212298 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69269ffe-3005-4849-abd1-97462a39286f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c916d419-mllrg\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.212508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.212356 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scm46\" (UniqueName: \"kubernetes.io/projected/69269ffe-3005-4849-abd1-97462a39286f-kube-api-access-scm46\") pod \"maas-api-revoke-sub-c916d419-mllrg\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.212508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.212467 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69269ffe-3005-4849-abd1-97462a39286f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c916d419-mllrg\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.213187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.213163 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69269ffe-3005-4849-abd1-97462a39286f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c916d419-mllrg\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.214880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.214854 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69269ffe-3005-4849-abd1-97462a39286f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c916d419-mllrg\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.274187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.274120 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scm46\" (UniqueName: \"kubernetes.io/projected/69269ffe-3005-4849-abd1-97462a39286f-kube-api-access-scm46\") pod \"maas-api-revoke-sub-c916d419-mllrg\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.344792 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.344756 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:10.593419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.593321 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" event={"ID":"a0b7814a-5fef-47e6-83a9-b1c511303a04","Type":"ContainerStarted","Data":"e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693"} Oct 31 00:28:10.732557 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:28:10.732524 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69269ffe_3005_4849_abd1_97462a39286f.slice/crio-ea3910c81035a8526412e9dd0ac3e1e76db7c80a2e1a999a9ceca8b437b16ddf WatchSource:0}: Error finding container ea3910c81035a8526412e9dd0ac3e1e76db7c80a2e1a999a9ceca8b437b16ddf: Status 404 returned error can't find the container with id ea3910c81035a8526412e9dd0ac3e1e76db7c80a2e1a999a9ceca8b437b16ddf Oct 31 00:28:10.771422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.771393 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg"] Oct 31 00:28:10.810434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.810397 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:28:10.810684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.810659 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 00:28:10.810787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:10.810729 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 00:28:11.612115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:11.612069 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" event={"ID":"69269ffe-3005-4849-abd1-97462a39286f","Type":"ContainerStarted","Data":"da5fe9c2fde4acf0532144cf6ed979ae3b980667ca538eca8f4e633f579cde51"} Oct 31 00:28:11.612778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:11.612755 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" event={"ID":"69269ffe-3005-4849-abd1-97462a39286f","Type":"ContainerStarted","Data":"ea3910c81035a8526412e9dd0ac3e1e76db7c80a2e1a999a9ceca8b437b16ddf"} Oct 31 00:28:11.729793 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:11.729741 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" podStartSLOduration=2.729727967 podStartE2EDuration="2.729727967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:11.727964955 +0000 UTC m=+1524.203323379" watchObservedRunningTime="2026-10-31 00:28:11.729727967 +0000 UTC m=+1524.205086409" Oct 31 00:28:11.810161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:11.810128 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 00:28:11.810400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:11.810372 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 00:28:12.622037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:12.622002 4969 generic.go:356] "Generic (PLEG): container finished" podID="69269ffe-3005-4849-abd1-97462a39286f" containerID="da5fe9c2fde4acf0532144cf6ed979ae3b980667ca538eca8f4e633f579cde51" exitCode=0 Oct 31 00:28:12.622523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:12.622083 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" event={"ID":"69269ffe-3005-4849-abd1-97462a39286f","Type":"ContainerDied","Data":"da5fe9c2fde4acf0532144cf6ed979ae3b980667ca538eca8f4e633f579cde51"} Oct 31 00:28:12.810159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:12.810128 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 00:28:12.810339 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:12.810199 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 00:28:13.812031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:13.811993 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 00:28:13.812664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:13.812625 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 00:28:14.355641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.355256 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:14.517243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.517145 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69269ffe-3005-4849-abd1-97462a39286f-kube-api-access-scm46\" (UniqueName: \"kubernetes.io/projected/69269ffe-3005-4849-abd1-97462a39286f-kube-api-access-scm46\") pod \"69269ffe-3005-4849-abd1-97462a39286f\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " Oct 31 00:28:14.517532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.517507 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69269ffe-3005-4849-abd1-97462a39286f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69269ffe-3005-4849-abd1-97462a39286f-maas-api-openshift-ca\") pod \"69269ffe-3005-4849-abd1-97462a39286f\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " Oct 31 00:28:14.517862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.517837 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69269ffe-3005-4849-abd1-97462a39286f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69269ffe-3005-4849-abd1-97462a39286f-maas-api-xks-ca\") pod \"69269ffe-3005-4849-abd1-97462a39286f\" (UID: \"69269ffe-3005-4849-abd1-97462a39286f\") " Oct 31 00:28:14.604622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.604292 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69269ffe-3005-4849-abd1-97462a39286f-maas-api-openshift-ca" pod "69269ffe-3005-4849-abd1-97462a39286f" (UID: "69269ffe-3005-4849-abd1-97462a39286f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:14.605855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.605806 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69269ffe-3005-4849-abd1-97462a39286f-maas-api-xks-ca" pod "69269ffe-3005-4849-abd1-97462a39286f" (UID: "69269ffe-3005-4849-abd1-97462a39286f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:14.690620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.665684 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69269ffe-3005-4849-abd1-97462a39286f-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.694627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.691340 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69269ffe-3005-4849-abd1-97462a39286f-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.736660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.729817 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69269ffe-3005-4849-abd1-97462a39286f-kube-api-access-scm46" pod "69269ffe-3005-4849-abd1-97462a39286f" (UID: "69269ffe-3005-4849-abd1-97462a39286f"). InnerVolumeSpecName "kube-api-access-scm46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:14.824686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.823950 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 00:28:14.824686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.824011 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 00:28:14.824686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.824658 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scm46\" (UniqueName: \"kubernetes.io/projected/69269ffe-3005-4849-abd1-97462a39286f-kube-api-access-scm46\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:28:15.004658 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.995426 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" event={"ID":"69269ffe-3005-4849-abd1-97462a39286f","Type":"ContainerDied","Data":"ea3910c81035a8526412e9dd0ac3e1e76db7c80a2e1a999a9ceca8b437b16ddf"} Oct 31 00:28:15.004658 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.995489 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea3910c81035a8526412e9dd0ac3e1e76db7c80a2e1a999a9ceca8b437b16ddf" Oct 31 00:28:15.004658 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:14.995559 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg" Oct 31 00:28:15.247620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:15.246692 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw" podUID="f8736325-6dcb-4b95-ac2c-a9a25635af0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:28:15.251624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:15.247895 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:28:15.435676 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:15.434819 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:28:15.560769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:15.560664 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:28:15.605940 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:15.605894 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:28:15.814196 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:15.814093 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 00:28:15.814481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:15.814433 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 00:28:16.814489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:16.814459 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:28:16.814953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:16.814935 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:28:16.815701 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:16.815677 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:28:22.872063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:22.872032 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m"] Oct 31 00:28:22.872682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:22.872668 4969 cpu_manager.go:414] "containerMap: removing container" podUID="69269ffe-3005-4849-abd1-97462a39286f" containerName="revoke-keys" Oct 31 00:28:22.872745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:22.872683 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69269ffe-3005-4849-abd1-97462a39286f" containerName="revoke-keys" Oct 31 00:28:22.872838 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:22.872828 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="69269ffe-3005-4849-abd1-97462a39286f" containerName="revoke-keys" Oct 31 00:28:22.875416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:22.875399 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:22.930759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:22.930721 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a4dc2e-openshift-default-dockercfg-92x4n\"" Oct 31 00:28:23.009463 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.009417 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-podinfo\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.020663 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.009836 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-certs\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.021063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.021037 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-data\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.021296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.021276 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2c8\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-kube-api-access-kp2c8\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.021447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.021432 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-socket\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.021673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.021644 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439b8775-aa5e-4faa-b117-08be2dfe0d00-istiod-ca-cert\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.021833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.021815 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-envoy\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.021984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.021970 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-token\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.022074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.022064 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-credential-socket\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.029673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.029629 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m"] Oct 31 00:28:23.124068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.123975 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-envoy\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124046 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-token\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124069 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-credential-socket\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124114 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-podinfo\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124184 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-certs\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124255 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-data\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124320 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2c8\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-kube-api-access-kp2c8\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124385 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-socket\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124466 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439b8775-aa5e-4faa-b117-08be2dfe0d00-istiod-ca-cert\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124503 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-credential-socket\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124653 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-data\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124781 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-socket\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.124891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.124857 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-certs\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.125316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.125291 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439b8775-aa5e-4faa-b117-08be2dfe0d00-istiod-ca-cert\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.126579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.126552 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-envoy\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.127577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.127556 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-podinfo\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.185789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.185756 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-token\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.186002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.185943 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2c8\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-kube-api-access-kp2c8\") pod \"e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.187774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.187747 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:23.714058 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:28:23.714019 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439b8775_aa5e_4faa_b117_08be2dfe0d00.slice/crio-4f6da55602aa10fef9ad0183a6553fe04c5d43918d2460666d098d277c45caba WatchSource:0}: Error finding container 4f6da55602aa10fef9ad0183a6553fe04c5d43918d2460666d098d277c45caba: Status 404 returned error can't find the container with id 4f6da55602aa10fef9ad0183a6553fe04c5d43918d2460666d098d277c45caba Oct 31 00:28:23.714399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.714370 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m"] Oct 31 00:28:23.716922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.716876 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:28:23.717040 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.716992 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:28:23.717105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:23.717040 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:28:24.078346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:24.078249 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" event={"ID":"439b8775-aa5e-4faa-b117-08be2dfe0d00","Type":"ContainerStarted","Data":"72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a"} Oct 31 00:28:24.078346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:24.078306 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" event={"ID":"439b8775-aa5e-4faa-b117-08be2dfe0d00","Type":"ContainerStarted","Data":"4f6da55602aa10fef9ad0183a6553fe04c5d43918d2460666d098d277c45caba"} Oct 31 00:28:24.188713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:24.188685 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:24.193625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:24.193577 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:24.203176 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:24.203129 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" podStartSLOduration=2.203116326 podStartE2EDuration="2.203116326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:24.197692185 +0000 UTC m=+1536.673050608" watchObservedRunningTime="2026-10-31 00:28:24.203116326 +0000 UTC m=+1536.678474748" Oct 31 00:28:25.091639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:25.091585 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:25.097227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:25.097199 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:28:32.736697 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:28:32.736623 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:28:48.277370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:48.277326 4969 scope.go:122] "RemoveContainer" containerID="e2baebddd445f9aa8f876cce8b85f9a76978322542ba97c3eb4a0190447262e9" Oct 31 00:28:58.951018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:58.950925 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z"] Oct 31 00:28:58.956764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:58.956740 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.009023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.008989 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z"] Oct 31 00:28:59.051306 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.051278 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:28:59.051452 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.051343 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:28:59.107954 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.107912 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bg5l\" (UniqueName: \"kubernetes.io/projected/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-kube-api-access-5bg5l\") pod \"maas-api-revoke-sub-bfead6dd-czl5z\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.108112 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.108040 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfead6dd-czl5z\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.108112 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.108095 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfead6dd-czl5z\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.209648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.209546 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bg5l\" (UniqueName: \"kubernetes.io/projected/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-kube-api-access-5bg5l\") pod \"maas-api-revoke-sub-bfead6dd-czl5z\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.209778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.209738 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfead6dd-czl5z\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.209837 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.209801 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfead6dd-czl5z\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.210477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.210456 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfead6dd-czl5z\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.212295 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.212278 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfead6dd-czl5z\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.270288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.270257 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bg5l\" (UniqueName: \"kubernetes.io/projected/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-kube-api-access-5bg5l\") pod \"maas-api-revoke-sub-bfead6dd-czl5z\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.457861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.457828 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn"] Oct 31 00:28:59.461137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.461080 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.515582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.515555 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-835921-openshift-default-dockercfg-549j9\"" Oct 31 00:28:59.523080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.523053 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn"] Oct 31 00:28:59.569652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.569623 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:28:59.620019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.619985 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-envoy\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.620185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.620040 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcj2p\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-kube-api-access-fcj2p\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.620185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.620072 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-credential-socket\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.620271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.620251 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-socket\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.620759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.620283 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-podinfo\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.620759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.620301 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-token\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.620759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.620436 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-data\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.620759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.620496 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-certs\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.620759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.620541 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6a1ab-f335-42fc-a645-b18f8d554ead-istiod-ca-cert\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.730709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.728873 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-data\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.730709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.729033 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-certs\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.730709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.729074 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6a1ab-f335-42fc-a645-b18f8d554ead-istiod-ca-cert\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.808667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.794052 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-envoy\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.808667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.794139 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcj2p\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-kube-api-access-fcj2p\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.808667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.794174 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-credential-socket\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.808667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.794355 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-socket\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.808667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.794387 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-podinfo\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.808667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.794413 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-token\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.808667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.795649 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6a1ab-f335-42fc-a645-b18f8d554ead-istiod-ca-cert\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.808667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.795940 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-data\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.808667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.796163 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-certs\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.845795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.836172 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-credential-socket\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.845795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.836865 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-socket\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.847652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.846730 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-podinfo\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.850481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.850252 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-envoy\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.879646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.878625 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-token\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.898312 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.898282 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcj2p\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-kube-api-access-fcj2p\") pod \"e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:28:59.985872 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:28:59.985845 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a61db72_53a7_4ab0_8047_4bbcdc86b84c.slice/crio-9be8879563fbcafcb641e9cfef0eaefa87a86a2997c3ee8d7f6f1bda2afb6cd3 WatchSource:0}: Error finding container 9be8879563fbcafcb641e9cfef0eaefa87a86a2997c3ee8d7f6f1bda2afb6cd3: Status 404 returned error can't find the container with id 9be8879563fbcafcb641e9cfef0eaefa87a86a2997c3ee8d7f6f1bda2afb6cd3 Oct 31 00:28:59.986767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:28:59.986744 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z"] Oct 31 00:29:00.074439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:00.074411 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:29:00.436048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:00.436009 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" event={"ID":"7a61db72-53a7-4ab0-8047-4bbcdc86b84c","Type":"ContainerStarted","Data":"081d774a1a1d3123b4ef0b42458dfab5f7e95ee5c6104b46af6146b07bc213d8"} Oct 31 00:29:00.436222 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:00.436067 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" event={"ID":"7a61db72-53a7-4ab0-8047-4bbcdc86b84c","Type":"ContainerStarted","Data":"9be8879563fbcafcb641e9cfef0eaefa87a86a2997c3ee8d7f6f1bda2afb6cd3"} Oct 31 00:29:00.507365 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:29:00.507331 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e6a1ab_f335_42fc_a645_b18f8d554ead.slice/crio-5fbacdee4709d5df5b5776144f220d4b11cc44f2d2ed20bb2f0a9216f4740247 WatchSource:0}: Error finding container 5fbacdee4709d5df5b5776144f220d4b11cc44f2d2ed20bb2f0a9216f4740247: Status 404 returned error can't find the container with id 5fbacdee4709d5df5b5776144f220d4b11cc44f2d2ed20bb2f0a9216f4740247 Oct 31 00:29:00.508483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:00.508457 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn"] Oct 31 00:29:00.509956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:00.509924 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:00.510048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:00.509990 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:00.510048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:00.510020 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:00.556532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:00.556475 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" podStartSLOduration=2.556461797 podStartE2EDuration="2.556461797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:00.552053247 +0000 UTC m=+1573.027411671" watchObservedRunningTime="2026-10-31 00:29:00.556461797 +0000 UTC m=+1573.031820249" Oct 31 00:29:01.445195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:01.445162 4969 generic.go:356] "Generic (PLEG): container finished" podID="7a61db72-53a7-4ab0-8047-4bbcdc86b84c" containerID="081d774a1a1d3123b4ef0b42458dfab5f7e95ee5c6104b46af6146b07bc213d8" exitCode=0 Oct 31 00:29:01.445623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:01.445241 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" event={"ID":"7a61db72-53a7-4ab0-8047-4bbcdc86b84c","Type":"ContainerDied","Data":"081d774a1a1d3123b4ef0b42458dfab5f7e95ee5c6104b46af6146b07bc213d8"} Oct 31 00:29:01.447233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:01.447210 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" event={"ID":"05e6a1ab-f335-42fc-a645-b18f8d554ead","Type":"ContainerStarted","Data":"1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82"} Oct 31 00:29:01.447342 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:01.447247 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" event={"ID":"05e6a1ab-f335-42fc-a645-b18f8d554ead","Type":"ContainerStarted","Data":"5fbacdee4709d5df5b5776144f220d4b11cc44f2d2ed20bb2f0a9216f4740247"} Oct 31 00:29:01.685107 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:01.685055 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" podStartSLOduration=2.685040467 podStartE2EDuration="2.685040467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:01.681147537 +0000 UTC m=+1574.156505959" watchObservedRunningTime="2026-10-31 00:29:01.685040467 +0000 UTC m=+1574.160398889" Oct 31 00:29:02.074894 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.074863 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:29:02.080348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.080316 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:29:02.454987 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.454951 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:29:02.455934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.455915 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:29:02.654697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.654660 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:29:02.728692 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:29:02.728580 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:29:02.838728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.838698 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-xks-ca\") pod \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " Oct 31 00:29:02.838924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.838737 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-kube-api-access-5bg5l\" (UniqueName: \"kubernetes.io/projected/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-kube-api-access-5bg5l\") pod \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " Oct 31 00:29:02.838924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.838800 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-openshift-ca\") pod \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\" (UID: \"7a61db72-53a7-4ab0-8047-4bbcdc86b84c\") " Oct 31 00:29:02.839341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.839312 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-openshift-ca" pod "7a61db72-53a7-4ab0-8047-4bbcdc86b84c" (UID: "7a61db72-53a7-4ab0-8047-4bbcdc86b84c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:02.841285 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.841263 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-xks-ca" pod "7a61db72-53a7-4ab0-8047-4bbcdc86b84c" (UID: "7a61db72-53a7-4ab0-8047-4bbcdc86b84c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:02.841528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.841507 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-kube-api-access-5bg5l" pod "7a61db72-53a7-4ab0-8047-4bbcdc86b84c" (UID: "7a61db72-53a7-4ab0-8047-4bbcdc86b84c"). InnerVolumeSpecName "kube-api-access-5bg5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:02.940457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.940417 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:29:02.940457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.940447 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bg5l\" (UniqueName: \"kubernetes.io/projected/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-kube-api-access-5bg5l\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:29:02.940457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:02.940459 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a61db72-53a7-4ab0-8047-4bbcdc86b84c-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.466241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:03.466209 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" Oct 31 00:29:03.466779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:03.466243 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z" event={"ID":"7a61db72-53a7-4ab0-8047-4bbcdc86b84c","Type":"ContainerDied","Data":"9be8879563fbcafcb641e9cfef0eaefa87a86a2997c3ee8d7f6f1bda2afb6cd3"} Oct 31 00:29:03.466779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:03.466294 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9be8879563fbcafcb641e9cfef0eaefa87a86a2997c3ee8d7f6f1bda2afb6cd3" Oct 31 00:29:09.324480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.324445 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6"] Oct 31 00:29:09.325641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.325167 4969 cpu_manager.go:414] "containerMap: removing container" podUID="7a61db72-53a7-4ab0-8047-4bbcdc86b84c" containerName="revoke-keys" Oct 31 00:29:09.325641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.325185 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a61db72-53a7-4ab0-8047-4bbcdc86b84c" containerName="revoke-keys" Oct 31 00:29:09.325641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.325302 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a61db72-53a7-4ab0-8047-4bbcdc86b84c" containerName="revoke-keys" Oct 31 00:29:09.331387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.327874 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:09.374089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.374059 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6"] Oct 31 00:29:09.416514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.416483 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-835921\"" Oct 31 00:29:09.416680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.416552 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-835921-dockercfg-s79h6\"" Oct 31 00:29:09.438233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.438204 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ee07e7-f923-43eb-b1a4-aec71c96013f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6\" (UID: \"98ee07e7-f923-43eb-b1a4-aec71c96013f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:09.438385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.438370 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfn6n\" (UniqueName: \"kubernetes.io/projected/98ee07e7-f923-43eb-b1a4-aec71c96013f-kube-api-access-rfn6n\") pod \"payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6\" (UID: \"98ee07e7-f923-43eb-b1a4-aec71c96013f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:09.540207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.540170 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ee07e7-f923-43eb-b1a4-aec71c96013f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6\" (UID: \"98ee07e7-f923-43eb-b1a4-aec71c96013f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:09.540378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.540345 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfn6n\" (UniqueName: \"kubernetes.io/projected/98ee07e7-f923-43eb-b1a4-aec71c96013f-kube-api-access-rfn6n\") pod \"payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6\" (UID: \"98ee07e7-f923-43eb-b1a4-aec71c96013f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:09.541002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.540984 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ee07e7-f923-43eb-b1a4-aec71c96013f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6\" (UID: \"98ee07e7-f923-43eb-b1a4-aec71c96013f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:09.601438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.601368 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfn6n\" (UniqueName: \"kubernetes.io/projected/98ee07e7-f923-43eb-b1a4-aec71c96013f-kube-api-access-rfn6n\") pod \"payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6\" (UID: \"98ee07e7-f923-43eb-b1a4-aec71c96013f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:09.643039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:09.643016 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:10.031611 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:29:10.031568 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ee07e7_f923_43eb_b1a4_aec71c96013f.slice/crio-cf3ffd11acc97eb9f1cc1424b425eeb1d5c0916ff9951db356a4b2f3a24a761d WatchSource:0}: Error finding container cf3ffd11acc97eb9f1cc1424b425eeb1d5c0916ff9951db356a4b2f3a24a761d: Status 404 returned error can't find the container with id cf3ffd11acc97eb9f1cc1424b425eeb1d5c0916ff9951db356a4b2f3a24a761d Oct 31 00:29:10.032007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:10.031985 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6"] Oct 31 00:29:10.536442 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:10.536411 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" event={"ID":"98ee07e7-f923-43eb-b1a4-aec71c96013f","Type":"ContainerStarted","Data":"b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831"} Oct 31 00:29:10.536951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:10.536462 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" event={"ID":"98ee07e7-f923-43eb-b1a4-aec71c96013f","Type":"ContainerStarted","Data":"cf3ffd11acc97eb9f1cc1424b425eeb1d5c0916ff9951db356a4b2f3a24a761d"} Oct 31 00:29:10.536951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:10.536515 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:10.654856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:10.654799 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" podStartSLOduration=1.6547842400000001 podStartE2EDuration="1.65478424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:10.651943865 +0000 UTC m=+1583.127302290" watchObservedRunningTime="2026-10-31 00:29:10.65478424 +0000 UTC m=+1583.130142662" Oct 31 00:29:21.548382 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:21.548355 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:29:32.732617 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:29:32.732558 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:29:35.855340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:35.855305 4969 generic.go:356] "Generic (PLEG): container finished" podID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerID="72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a" exitCode=0 Oct 31 00:29:35.855907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:35.855355 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" event={"ID":"439b8775-aa5e-4faa-b117-08be2dfe0d00","Type":"ContainerDied","Data":"72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a"} Oct 31 00:29:35.855907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:35.855855 4969 scope.go:122] "RemoveContainer" containerID="72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a" Oct 31 00:29:35.857416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:35.857387 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:35.857506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:35.857453 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:35.857506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:35.857503 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:36.868148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:36.868114 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" event={"ID":"439b8775-aa5e-4faa-b117-08be2dfe0d00","Type":"ContainerStarted","Data":"19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4"} Oct 31 00:29:37.198676 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:37.191978 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:29:37.202520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:37.202476 4969 patch_prober.go:29] interesting pod/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:29:37.202722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:37.202547 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 00:29:38.192618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:38.188975 4969 patch_prober.go:29] interesting pod/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:29:38.192618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:38.189035 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 00:29:39.204061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:39.204023 4969 patch_prober.go:29] interesting pod/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:29:39.204474 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:39.204094 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 00:29:40.096746 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:40.096707 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:29:40.200619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:40.195037 4969 patch_prober.go:29] interesting pod/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:29:40.200619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:40.195108 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 00:29:41.201625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:41.197036 4969 patch_prober.go:29] interesting pod/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 00:29:41.201625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:41.197109 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 00:29:42.193078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:42.193048 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:29:42.194223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:42.194204 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:29:46.045224 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.045191 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss"] Oct 31 00:29:46.049990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.049973 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.090136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.090108 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbzm\" (UniqueName: \"kubernetes.io/projected/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-kube-api-access-mpbzm\") pod \"maas-api-revoke-sub-d19ef0b3-ptjss\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.090264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.090151 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d19ef0b3-ptjss\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.090367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.090348 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d19ef0b3-ptjss\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.106439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.106411 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss"] Oct 31 00:29:46.150497 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.150466 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:46.150690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.150578 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:29:46.191651 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.191618 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbzm\" (UniqueName: \"kubernetes.io/projected/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-kube-api-access-mpbzm\") pod \"maas-api-revoke-sub-d19ef0b3-ptjss\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.191804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.191661 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d19ef0b3-ptjss\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.191804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.191778 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d19ef0b3-ptjss\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.192326 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.192297 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d19ef0b3-ptjss\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.194426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.194402 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d19ef0b3-ptjss\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.252858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.252829 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbzm\" (UniqueName: \"kubernetes.io/projected/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-kube-api-access-mpbzm\") pod \"maas-api-revoke-sub-d19ef0b3-ptjss\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.363381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.363296 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:46.759207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.759137 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss"] Oct 31 00:29:46.974020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:46.973979 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" event={"ID":"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e","Type":"ContainerStarted","Data":"9fb540905805ac61a38d9d865084647b2e7c676909009977913310977f474493"} Oct 31 00:29:47.984072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:47.983985 4969 generic.go:356] "Generic (PLEG): container finished" podID="91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e" containerID="b85686e9e04744deda1cde25ed086de105e61965dae1f6115b3f3f3fc1c738fc" exitCode=0 Oct 31 00:29:47.984072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:47.984038 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" event={"ID":"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e","Type":"ContainerDied","Data":"b85686e9e04744deda1cde25ed086de105e61965dae1f6115b3f3f3fc1c738fc"} Oct 31 00:29:49.174398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.174368 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:49.240923 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.240890 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-xks-ca\") pod \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " Oct 31 00:29:49.241108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.240952 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-openshift-ca\") pod \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " Oct 31 00:29:49.241186 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.241115 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-kube-api-access-mpbzm\" (UniqueName: \"kubernetes.io/projected/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-kube-api-access-mpbzm\") pod \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\" (UID: \"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e\") " Oct 31 00:29:49.242088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.241872 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-openshift-ca" pod "91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e" (UID: "91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:49.244004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.243882 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-xks-ca" pod "91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e" (UID: "91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:49.244314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.244289 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-kube-api-access-mpbzm" pod "91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e" (UID: "91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e"). InnerVolumeSpecName "kube-api-access-mpbzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:49.343567 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.343530 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpbzm\" (UniqueName: \"kubernetes.io/projected/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-kube-api-access-mpbzm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:29:49.343567 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.343559 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:29:49.343567 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.343569 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:29:49.936218 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.936184 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj"] Oct 31 00:29:49.936947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.936931 4969 cpu_manager.go:414] "containerMap: removing container" podUID="91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e" containerName="revoke-keys" Oct 31 00:29:49.937007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.936949 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e" containerName="revoke-keys" Oct 31 00:29:49.937068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.937059 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e" containerName="revoke-keys" Oct 31 00:29:49.940852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:49.940834 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.035677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.018621 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2ccdde-openshift-default-dockercfg-rjnhf\"" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.051828 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-data\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.051877 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-podinfo\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.051921 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-certs\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.051973 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-socket\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.052018 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfdec536-f521-4f73-b2ef-8d421a1a688b-istiod-ca-cert\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.052144 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-token\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.052230 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-envoy\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.052278 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-credential-socket\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.052303 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5zrc\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-kube-api-access-d5zrc\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.063616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.060484 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj"] Oct 31 00:29:50.129636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.129579 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" event={"ID":"91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e","Type":"ContainerDied","Data":"9fb540905805ac61a38d9d865084647b2e7c676909009977913310977f474493"} Oct 31 00:29:50.129912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.129890 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fb540905805ac61a38d9d865084647b2e7c676909009977913310977f474493" Oct 31 00:29:50.130172 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.129696 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss" Oct 31 00:29:50.153630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.153576 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-certs\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.153782 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.153654 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-socket\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.153782 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.153690 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfdec536-f521-4f73-b2ef-8d421a1a688b-istiod-ca-cert\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.153898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.153798 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-token\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.153898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.153876 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-envoy\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.154005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.153925 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-credential-socket\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.154005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.153950 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5zrc\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-kube-api-access-d5zrc\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.154121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.154051 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-data\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.154121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.154069 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-podinfo\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.154121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.154101 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-certs\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.154272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.154128 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-socket\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.154620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.154548 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfdec536-f521-4f73-b2ef-8d421a1a688b-istiod-ca-cert\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.154729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.154647 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-credential-socket\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.154983 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.154963 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-data\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.157141 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.157118 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-podinfo\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.157223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.157159 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-envoy\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.219220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.219179 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-token\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.219619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.219281 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5zrc\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-kube-api-access-d5zrc\") pod \"e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.255283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.255250 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:50.710236 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:29:50.710185 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfdec536_f521_4f73_b2ef_8d421a1a688b.slice/crio-7a0b43bea1fcb9714a5b92223413f8c74a6641a1c836f7175a08260c7d15e4fb WatchSource:0}: Error finding container 7a0b43bea1fcb9714a5b92223413f8c74a6641a1c836f7175a08260c7d15e4fb: Status 404 returned error can't find the container with id 7a0b43bea1fcb9714a5b92223413f8c74a6641a1c836f7175a08260c7d15e4fb Oct 31 00:29:50.710992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.710968 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj"] Oct 31 00:29:50.714082 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.714045 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:50.714202 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.714124 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:50.714202 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:50.714170 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:51.145061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:51.145019 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" event={"ID":"dfdec536-f521-4f73-b2ef-8d421a1a688b","Type":"ContainerStarted","Data":"dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939"} Oct 31 00:29:51.145287 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:51.145090 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" event={"ID":"dfdec536-f521-4f73-b2ef-8d421a1a688b","Type":"ContainerStarted","Data":"7a0b43bea1fcb9714a5b92223413f8c74a6641a1c836f7175a08260c7d15e4fb"} Oct 31 00:29:51.255786 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:51.255752 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:51.257867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:51.257837 4969 patch_prober.go:29] interesting pod/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:29:51.258093 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:51.257901 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 00:29:51.275641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:51.268409 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" podStartSLOduration=2.268390219 podStartE2EDuration="2.268390219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:51.266068952 +0000 UTC m=+1623.741427379" watchObservedRunningTime="2026-10-31 00:29:51.268390219 +0000 UTC m=+1623.743748648" Oct 31 00:29:52.261581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:52.261548 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:53.163779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:53.163745 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:53.164852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:53.164833 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:29:57.953620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:57.953186 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq"] Oct 31 00:29:57.971625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:57.960064 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:29:58.071045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.070999 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2ccdde\"" Oct 31 00:29:58.072162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.072131 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2ccdde-dockercfg-22jwj\"" Oct 31 00:29:58.094253 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.094214 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq"] Oct 31 00:29:58.108140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.108101 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xqd\" (UniqueName: \"kubernetes.io/projected/29465fcc-6627-4e1b-96db-5c9f28b9f57e-kube-api-access-b8xqd\") pod \"payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq\" (UID: \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:29:58.108320 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.108214 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29465fcc-6627-4e1b-96db-5c9f28b9f57e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq\" (UID: \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:29:58.210100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.210014 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xqd\" (UniqueName: \"kubernetes.io/projected/29465fcc-6627-4e1b-96db-5c9f28b9f57e-kube-api-access-b8xqd\") pod \"payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq\" (UID: \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:29:58.210100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.210085 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29465fcc-6627-4e1b-96db-5c9f28b9f57e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq\" (UID: \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:29:58.210836 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.210810 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29465fcc-6627-4e1b-96db-5c9f28b9f57e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq\" (UID: \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:29:58.271010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.270986 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xqd\" (UniqueName: \"kubernetes.io/projected/29465fcc-6627-4e1b-96db-5c9f28b9f57e-kube-api-access-b8xqd\") pod \"payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq\" (UID: \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:29:58.340037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.340002 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:29:58.719943 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:29:58.719917 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29465fcc_6627_4e1b_96db_5c9f28b9f57e.slice/crio-ab43b884dfb3c7d7125eeb77768aa09c6883785974e5b4e5da574024ccbd6f8d WatchSource:0}: Error finding container ab43b884dfb3c7d7125eeb77768aa09c6883785974e5b4e5da574024ccbd6f8d: Status 404 returned error can't find the container with id ab43b884dfb3c7d7125eeb77768aa09c6883785974e5b4e5da574024ccbd6f8d Oct 31 00:29:58.721711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:58.721686 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq"] Oct 31 00:29:59.220137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:59.220097 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" event={"ID":"29465fcc-6627-4e1b-96db-5c9f28b9f57e","Type":"ContainerStarted","Data":"c4065fa63d2f418b4f01942951286656a8b739773fdc264805dd3253fe94af46"} Oct 31 00:29:59.220647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:59.220168 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" event={"ID":"29465fcc-6627-4e1b-96db-5c9f28b9f57e","Type":"ContainerStarted","Data":"ab43b884dfb3c7d7125eeb77768aa09c6883785974e5b4e5da574024ccbd6f8d"} Oct 31 00:29:59.220647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:59.220204 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:29:59.336125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:29:59.336066 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" podStartSLOduration=2.336047549 podStartE2EDuration="2.336047549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:59.33489132 +0000 UTC m=+1631.810249745" watchObservedRunningTime="2026-10-31 00:29:59.336047549 +0000 UTC m=+1631.811405974" Oct 31 00:30:00.177821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.177790 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b"] Oct 31 00:30:00.182390 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.182369 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:00.183101 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.183078 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp"] Oct 31 00:30:00.187294 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.187274 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:00.205489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.205458 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw"] Oct 31 00:30:00.209813 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.209792 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:00.234333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.234307 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv"] Oct 31 00:30:00.236548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.236512 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:30:00.238884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.238865 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:00.239819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.239795 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s"] Oct 31 00:30:00.244158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.244139 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:00.248752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.248511 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npzs7\" (UniqueName: \"kubernetes.io/projected/ca188ff5-a199-45e7-83cc-78a5d975b140-kube-api-access-npzs7\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b\" (UID: \"ca188ff5-a199-45e7-83cc-78a5d975b140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:00.248752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.248564 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca188ff5-a199-45e7-83cc-78a5d975b140-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b\" (UID: \"ca188ff5-a199-45e7-83cc-78a5d975b140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:00.294574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.287206 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b"] Oct 31 00:30:00.294574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.293582 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp"] Oct 31 00:30:00.303380 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.303349 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw"] Oct 31 00:30:00.328652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.328581 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s"] Oct 31 00:30:00.350311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.350271 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npzs7\" (UniqueName: \"kubernetes.io/projected/ca188ff5-a199-45e7-83cc-78a5d975b140-kube-api-access-npzs7\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b\" (UID: \"ca188ff5-a199-45e7-83cc-78a5d975b140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:00.350514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.350330 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcx2f\" (UniqueName: \"kubernetes.io/projected/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-kube-api-access-lcx2f\") pod \"maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw\" (UID: \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:00.350514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.350371 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a77832-b33e-4306-8f05-2e034730ebed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp\" (UID: \"17a77832-b33e-4306-8f05-2e034730ebed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:00.350514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.350423 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca188ff5-a199-45e7-83cc-78a5d975b140-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b\" (UID: \"ca188ff5-a199-45e7-83cc-78a5d975b140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:00.350514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.350492 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw\" (UID: \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:00.350762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.350548 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv485\" (UniqueName: \"kubernetes.io/projected/17a77832-b33e-4306-8f05-2e034730ebed-kube-api-access-pv485\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp\" (UID: \"17a77832-b33e-4306-8f05-2e034730ebed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:00.350997 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.350971 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qd5k\" (UniqueName: \"kubernetes.io/projected/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-kube-api-access-4qd5k\") pod \"maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv\" (UID: \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:00.351104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.351021 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3714929-2dbd-4696-b871-d25966e1af84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s\" (UID: \"f3714929-2dbd-4696-b871-d25966e1af84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:00.351167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.351143 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdq6\" (UniqueName: \"kubernetes.io/projected/f3714929-2dbd-4696-b871-d25966e1af84-kube-api-access-qpdq6\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s\" (UID: \"f3714929-2dbd-4696-b871-d25966e1af84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:00.351245 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.351224 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv\" (UID: \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:00.351393 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.351369 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca188ff5-a199-45e7-83cc-78a5d975b140-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b\" (UID: \"ca188ff5-a199-45e7-83cc-78a5d975b140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:00.358931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.358899 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv"] Oct 31 00:30:00.411534 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.411503 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npzs7\" (UniqueName: \"kubernetes.io/projected/ca188ff5-a199-45e7-83cc-78a5d975b140-kube-api-access-npzs7\") pod \"maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b\" (UID: \"ca188ff5-a199-45e7-83cc-78a5d975b140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:00.452561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.452462 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qd5k\" (UniqueName: \"kubernetes.io/projected/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-kube-api-access-4qd5k\") pod \"maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv\" (UID: \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:00.452561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.452509 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3714929-2dbd-4696-b871-d25966e1af84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s\" (UID: \"f3714929-2dbd-4696-b871-d25966e1af84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:00.452561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.452559 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdq6\" (UniqueName: \"kubernetes.io/projected/f3714929-2dbd-4696-b871-d25966e1af84-kube-api-access-qpdq6\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s\" (UID: \"f3714929-2dbd-4696-b871-d25966e1af84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:00.452882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.452619 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv\" (UID: \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:00.452882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.452696 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcx2f\" (UniqueName: \"kubernetes.io/projected/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-kube-api-access-lcx2f\") pod \"maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw\" (UID: \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:00.452882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.452717 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a77832-b33e-4306-8f05-2e034730ebed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp\" (UID: \"17a77832-b33e-4306-8f05-2e034730ebed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:00.452882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.452779 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw\" (UID: \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:00.452882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.452814 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv485\" (UniqueName: \"kubernetes.io/projected/17a77832-b33e-4306-8f05-2e034730ebed-kube-api-access-pv485\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp\" (UID: \"17a77832-b33e-4306-8f05-2e034730ebed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:00.453412 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.453384 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3714929-2dbd-4696-b871-d25966e1af84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s\" (UID: \"f3714929-2dbd-4696-b871-d25966e1af84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:00.453559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.453535 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv\" (UID: \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:00.453671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.453646 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a77832-b33e-4306-8f05-2e034730ebed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp\" (UID: \"17a77832-b33e-4306-8f05-2e034730ebed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:00.453756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.453734 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw\" (UID: \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:00.500692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.500655 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:00.517051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.517020 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdq6\" (UniqueName: \"kubernetes.io/projected/f3714929-2dbd-4696-b871-d25966e1af84-kube-api-access-qpdq6\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s\" (UID: \"f3714929-2dbd-4696-b871-d25966e1af84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:00.517284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.517265 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv485\" (UniqueName: \"kubernetes.io/projected/17a77832-b33e-4306-8f05-2e034730ebed-kube-api-access-pv485\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp\" (UID: \"17a77832-b33e-4306-8f05-2e034730ebed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:00.517437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.517418 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcx2f\" (UniqueName: \"kubernetes.io/projected/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-kube-api-access-lcx2f\") pod \"maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw\" (UID: \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:00.517550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.517526 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qd5k\" (UniqueName: \"kubernetes.io/projected/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-kube-api-access-4qd5k\") pod \"maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv\" (UID: \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:00.524291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.524260 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:00.558029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.557998 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:00.566662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.565552 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:00.808158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.808078 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:00.900025 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:00.899998 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca188ff5_a199_45e7_83cc_78a5d975b140.slice/crio-7a1ff023f17268082916a37504d4b944405891e145610024177d2fdcccf3bde2 WatchSource:0}: Error finding container 7a1ff023f17268082916a37504d4b944405891e145610024177d2fdcccf3bde2: Status 404 returned error can't find the container with id 7a1ff023f17268082916a37504d4b944405891e145610024177d2fdcccf3bde2 Oct 31 00:30:00.903753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.903730 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b"] Oct 31 00:30:00.947821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.947790 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw"] Oct 31 00:30:00.989792 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.989764 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv"] Oct 31 00:30:00.998797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:00.998773 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s"] Oct 31 00:30:01.079746 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:01.079671 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79638dc_69ec_4f25_a8fa_a9baeba3e8fc.slice/crio-eba428aad3c7544226a2a10e523787d1f5e4aa9ca1178ac6bff75a8c9ba5b402 WatchSource:0}: Error finding container eba428aad3c7544226a2a10e523787d1f5e4aa9ca1178ac6bff75a8c9ba5b402: Status 404 returned error can't find the container with id eba428aad3c7544226a2a10e523787d1f5e4aa9ca1178ac6bff75a8c9ba5b402 Oct 31 00:30:01.080515 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:01.080488 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3714929_2dbd_4696_b871_d25966e1af84.slice/crio-11d38c6c6b8ea9c8b671c3d5b24df32bfaead86376b3423353e3a86856eb8162 WatchSource:0}: Error finding container 11d38c6c6b8ea9c8b671c3d5b24df32bfaead86376b3423353e3a86856eb8162: Status 404 returned error can't find the container with id 11d38c6c6b8ea9c8b671c3d5b24df32bfaead86376b3423353e3a86856eb8162 Oct 31 00:30:01.193416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:01.193382 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp"] Oct 31 00:30:01.237967 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:01.237931 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" event={"ID":"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc","Type":"ContainerStarted","Data":"eba428aad3c7544226a2a10e523787d1f5e4aa9ca1178ac6bff75a8c9ba5b402"} Oct 31 00:30:01.239761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:01.239732 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" event={"ID":"6d208cbe-da72-4e31-b46f-4ae0e3d994b0","Type":"ContainerStarted","Data":"2df8065caa345ee3fd8b251062069f8f3200d6cdac3fee6837b9e41a86f7aab1"} Oct 31 00:30:01.242606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:01.242558 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" event={"ID":"f3714929-2dbd-4696-b871-d25966e1af84","Type":"ContainerStarted","Data":"11d38c6c6b8ea9c8b671c3d5b24df32bfaead86376b3423353e3a86856eb8162"} Oct 31 00:30:01.244227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:01.244202 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" event={"ID":"ca188ff5-a199-45e7-83cc-78a5d975b140","Type":"ContainerStarted","Data":"7a1ff023f17268082916a37504d4b944405891e145610024177d2fdcccf3bde2"} Oct 31 00:30:01.281422 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:01.281372 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a77832_b33e_4306_8f05_2e034730ebed.slice/crio-cfea1f950e537fdcc88503fd188b68d7776e92a54b7ea9bcff8c719ca79c189a WatchSource:0}: Error finding container cfea1f950e537fdcc88503fd188b68d7776e92a54b7ea9bcff8c719ca79c189a: Status 404 returned error can't find the container with id cfea1f950e537fdcc88503fd188b68d7776e92a54b7ea9bcff8c719ca79c189a Oct 31 00:30:02.255511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.255430 4969 generic.go:356] "Generic (PLEG): container finished" podID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerID="bc1cd75a4c846d2eafa84ac49e87c7941433293b6395c9c3489ae76296b3cc1a" exitCode=60 Oct 31 00:30:02.255909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.255508 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" event={"ID":"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc","Type":"ContainerDied","Data":"bc1cd75a4c846d2eafa84ac49e87c7941433293b6395c9c3489ae76296b3cc1a"} Oct 31 00:30:02.255909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.255851 4969 scope.go:122] "RemoveContainer" containerID="bc1cd75a4c846d2eafa84ac49e87c7941433293b6395c9c3489ae76296b3cc1a" Oct 31 00:30:02.257427 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.257397 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" event={"ID":"17a77832-b33e-4306-8f05-2e034730ebed","Type":"ContainerStarted","Data":"aa7af23100759e386b270f9b343a993961bd8a8f70b2603c01932b36e1a716ed"} Oct 31 00:30:02.257527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.257454 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" event={"ID":"17a77832-b33e-4306-8f05-2e034730ebed","Type":"ContainerStarted","Data":"cfea1f950e537fdcc88503fd188b68d7776e92a54b7ea9bcff8c719ca79c189a"} Oct 31 00:30:02.259267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.259243 4969 generic.go:356] "Generic (PLEG): container finished" podID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerID="2c7073103eb8403682512c267b21e3ce11cba66075eabe854357cbede96ffdec" exitCode=60 Oct 31 00:30:02.259379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.259325 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" event={"ID":"6d208cbe-da72-4e31-b46f-4ae0e3d994b0","Type":"ContainerDied","Data":"2c7073103eb8403682512c267b21e3ce11cba66075eabe854357cbede96ffdec"} Oct 31 00:30:02.259535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.259517 4969 scope.go:122] "RemoveContainer" containerID="2c7073103eb8403682512c267b21e3ce11cba66075eabe854357cbede96ffdec" Oct 31 00:30:02.261466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.261446 4969 generic.go:356] "Generic (PLEG): container finished" podID="f3714929-2dbd-4696-b871-d25966e1af84" containerID="c0e50057913370b66f42b494edad67fa59a2388fd4bfec8668e79776f1b5c778" exitCode=60 Oct 31 00:30:02.261540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.261515 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" event={"ID":"f3714929-2dbd-4696-b871-d25966e1af84","Type":"ContainerDied","Data":"c0e50057913370b66f42b494edad67fa59a2388fd4bfec8668e79776f1b5c778"} Oct 31 00:30:02.261850 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.261833 4969 scope.go:122] "RemoveContainer" containerID="c0e50057913370b66f42b494edad67fa59a2388fd4bfec8668e79776f1b5c778" Oct 31 00:30:02.263761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.263739 4969 generic.go:356] "Generic (PLEG): container finished" podID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerID="2c400ff2b5acc0ddb28932f44af90110884660d512482e7bf6f351756b84462f" exitCode=60 Oct 31 00:30:02.263859 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.263811 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" event={"ID":"ca188ff5-a199-45e7-83cc-78a5d975b140","Type":"ContainerDied","Data":"2c400ff2b5acc0ddb28932f44af90110884660d512482e7bf6f351756b84462f"} Oct 31 00:30:02.263990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.263976 4969 scope.go:122] "RemoveContainer" containerID="2c400ff2b5acc0ddb28932f44af90110884660d512482e7bf6f351756b84462f" Oct 31 00:30:02.486577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:02.486508 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" podStartSLOduration=2.486489069 podStartE2EDuration="2.486489069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:02.485893621 +0000 UTC m=+1634.961252044" watchObservedRunningTime="2026-10-31 00:30:02.486489069 +0000 UTC m=+1634.961847493" Oct 31 00:30:02.729405 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:02.729369 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:30:03.274453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.274410 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" event={"ID":"6d208cbe-da72-4e31-b46f-4ae0e3d994b0","Type":"ContainerStarted","Data":"0ea7585577effac3956448d49d562aa55a537996f57f0bf41ace99aafabb3d7d"} Oct 31 00:30:03.276542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.276514 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" event={"ID":"f3714929-2dbd-4696-b871-d25966e1af84","Type":"ContainerStarted","Data":"5504334c6d63e13afa35bd4e6cb7a855b578a3ba278f19aca0e9f13bff712db6"} Oct 31 00:30:03.279833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.279721 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" event={"ID":"ca188ff5-a199-45e7-83cc-78a5d975b140","Type":"ContainerStarted","Data":"6c254016165a5aba8748c69c27fb5dd49afe46d05885b4edf8fd1b74dfcd7b5e"} Oct 31 00:30:03.283219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.283189 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" event={"ID":"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc","Type":"ContainerStarted","Data":"c8e272e9b0b6b5a62cc639e0b075ba36f2f2594e59cadd95575eb564d027fd9e"} Oct 31 00:30:03.285128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.285104 4969 generic.go:356] "Generic (PLEG): container finished" podID="17a77832-b33e-4306-8f05-2e034730ebed" containerID="aa7af23100759e386b270f9b343a993961bd8a8f70b2603c01932b36e1a716ed" exitCode=60 Oct 31 00:30:03.285248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.285155 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" event={"ID":"17a77832-b33e-4306-8f05-2e034730ebed","Type":"ContainerDied","Data":"aa7af23100759e386b270f9b343a993961bd8a8f70b2603c01932b36e1a716ed"} Oct 31 00:30:03.285514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.285494 4969 scope.go:122] "RemoveContainer" containerID="aa7af23100759e386b270f9b343a993961bd8a8f70b2603c01932b36e1a716ed" Oct 31 00:30:03.408517 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.408465 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" podStartSLOduration=3.408450866 podStartE2EDuration="3.408450866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.402667649 +0000 UTC m=+1635.878026072" watchObservedRunningTime="2026-10-31 00:30:03.408450866 +0000 UTC m=+1635.883809326" Oct 31 00:30:03.536956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.536907 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" podStartSLOduration=3.536887455 podStartE2EDuration="3.536887455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.535833564 +0000 UTC m=+1636.011191988" watchObservedRunningTime="2026-10-31 00:30:03.536887455 +0000 UTC m=+1636.012245878" Oct 31 00:30:03.656990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.656937 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" podStartSLOduration=3.656922534 podStartE2EDuration="3.656922534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.65050704 +0000 UTC m=+1636.125865463" watchObservedRunningTime="2026-10-31 00:30:03.656922534 +0000 UTC m=+1636.132281002" Oct 31 00:30:03.884713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:03.884663 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" podStartSLOduration=3.884648438 podStartE2EDuration="3.884648438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.879123483 +0000 UTC m=+1636.354481907" watchObservedRunningTime="2026-10-31 00:30:03.884648438 +0000 UTC m=+1636.360006860" Oct 31 00:30:04.295957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.295926 4969 generic.go:356] "Generic (PLEG): container finished" podID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerID="c8e272e9b0b6b5a62cc639e0b075ba36f2f2594e59cadd95575eb564d027fd9e" exitCode=60 Oct 31 00:30:04.296396 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.296000 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" event={"ID":"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc","Type":"ContainerDied","Data":"c8e272e9b0b6b5a62cc639e0b075ba36f2f2594e59cadd95575eb564d027fd9e"} Oct 31 00:30:04.296396 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.296046 4969 scope.go:122] "RemoveContainer" containerID="bc1cd75a4c846d2eafa84ac49e87c7941433293b6395c9c3489ae76296b3cc1a" Oct 31 00:30:04.296396 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.296376 4969 scope.go:122] "RemoveContainer" containerID="c8e272e9b0b6b5a62cc639e0b075ba36f2f2594e59cadd95575eb564d027fd9e" Oct 31 00:30:04.296834 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:04.296806 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv_odh-ai-gateway-infra(b79638dc-69ec-4f25-a8fa-a9baeba3e8fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" Oct 31 00:30:04.298084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.298059 4969 generic.go:356] "Generic (PLEG): container finished" podID="17a77832-b33e-4306-8f05-2e034730ebed" containerID="450260883f33b3224ef94349a0e758b5447079cccd9ab8bcff065aa3e59f56ac" exitCode=60 Oct 31 00:30:04.298205 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.298138 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" event={"ID":"17a77832-b33e-4306-8f05-2e034730ebed","Type":"ContainerDied","Data":"450260883f33b3224ef94349a0e758b5447079cccd9ab8bcff065aa3e59f56ac"} Oct 31 00:30:04.298445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.298431 4969 scope.go:122] "RemoveContainer" containerID="450260883f33b3224ef94349a0e758b5447079cccd9ab8bcff065aa3e59f56ac" Oct 31 00:30:04.298799 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:04.298776 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp_odh-ai-gateway-infra(17a77832-b33e-4306-8f05-2e034730ebed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" podUID="17a77832-b33e-4306-8f05-2e034730ebed" Oct 31 00:30:04.300048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.300028 4969 generic.go:356] "Generic (PLEG): container finished" podID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerID="0ea7585577effac3956448d49d562aa55a537996f57f0bf41ace99aafabb3d7d" exitCode=60 Oct 31 00:30:04.300148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.300046 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" event={"ID":"6d208cbe-da72-4e31-b46f-4ae0e3d994b0","Type":"ContainerDied","Data":"0ea7585577effac3956448d49d562aa55a537996f57f0bf41ace99aafabb3d7d"} Oct 31 00:30:04.300370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.300351 4969 scope.go:122] "RemoveContainer" containerID="0ea7585577effac3956448d49d562aa55a537996f57f0bf41ace99aafabb3d7d" Oct 31 00:30:04.300715 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:04.300690 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw_odh-ai-gateway-infra(6d208cbe-da72-4e31-b46f-4ae0e3d994b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" Oct 31 00:30:04.302300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.302281 4969 generic.go:356] "Generic (PLEG): container finished" podID="f3714929-2dbd-4696-b871-d25966e1af84" containerID="5504334c6d63e13afa35bd4e6cb7a855b578a3ba278f19aca0e9f13bff712db6" exitCode=60 Oct 31 00:30:04.302413 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.302367 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" event={"ID":"f3714929-2dbd-4696-b871-d25966e1af84","Type":"ContainerDied","Data":"5504334c6d63e13afa35bd4e6cb7a855b578a3ba278f19aca0e9f13bff712db6"} Oct 31 00:30:04.302575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.302560 4969 scope.go:122] "RemoveContainer" containerID="5504334c6d63e13afa35bd4e6cb7a855b578a3ba278f19aca0e9f13bff712db6" Oct 31 00:30:04.303102 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:04.303076 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s_odh-ai-gateway-infra(f3714929-2dbd-4696-b871-d25966e1af84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" podUID="f3714929-2dbd-4696-b871-d25966e1af84" Oct 31 00:30:04.304557 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.304538 4969 generic.go:356] "Generic (PLEG): container finished" podID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerID="6c254016165a5aba8748c69c27fb5dd49afe46d05885b4edf8fd1b74dfcd7b5e" exitCode=60 Oct 31 00:30:04.304674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.304641 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" event={"ID":"ca188ff5-a199-45e7-83cc-78a5d975b140","Type":"ContainerDied","Data":"6c254016165a5aba8748c69c27fb5dd49afe46d05885b4edf8fd1b74dfcd7b5e"} Oct 31 00:30:04.304872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.304855 4969 scope.go:122] "RemoveContainer" containerID="6c254016165a5aba8748c69c27fb5dd49afe46d05885b4edf8fd1b74dfcd7b5e" Oct 31 00:30:04.305146 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:04.305124 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b_odh-ai-gateway-infra(ca188ff5-a199-45e7-83cc-78a5d975b140)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" Oct 31 00:30:04.311647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.311626 4969 scope.go:122] "RemoveContainer" containerID="aa7af23100759e386b270f9b343a993961bd8a8f70b2603c01932b36e1a716ed" Oct 31 00:30:04.326541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.326518 4969 scope.go:122] "RemoveContainer" containerID="2c7073103eb8403682512c267b21e3ce11cba66075eabe854357cbede96ffdec" Oct 31 00:30:04.341056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.341035 4969 scope.go:122] "RemoveContainer" containerID="c0e50057913370b66f42b494edad67fa59a2388fd4bfec8668e79776f1b5c778" Oct 31 00:30:04.355495 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:04.355472 4969 scope.go:122] "RemoveContainer" containerID="2c400ff2b5acc0ddb28932f44af90110884660d512482e7bf6f351756b84462f" Oct 31 00:30:05.315091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:05.315061 4969 scope.go:122] "RemoveContainer" containerID="0ea7585577effac3956448d49d562aa55a537996f57f0bf41ace99aafabb3d7d" Oct 31 00:30:05.315541 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:05.315453 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw_odh-ai-gateway-infra(6d208cbe-da72-4e31-b46f-4ae0e3d994b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" Oct 31 00:30:05.317017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:05.316988 4969 scope.go:122] "RemoveContainer" containerID="5504334c6d63e13afa35bd4e6cb7a855b578a3ba278f19aca0e9f13bff712db6" Oct 31 00:30:05.317351 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:05.317326 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s_odh-ai-gateway-infra(f3714929-2dbd-4696-b871-d25966e1af84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" podUID="f3714929-2dbd-4696-b871-d25966e1af84" Oct 31 00:30:05.318688 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:05.318672 4969 scope.go:122] "RemoveContainer" containerID="6c254016165a5aba8748c69c27fb5dd49afe46d05885b4edf8fd1b74dfcd7b5e" Oct 31 00:30:05.319003 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:05.318979 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b_odh-ai-gateway-infra(ca188ff5-a199-45e7-83cc-78a5d975b140)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" Oct 31 00:30:05.320435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:05.320413 4969 scope.go:122] "RemoveContainer" containerID="c8e272e9b0b6b5a62cc639e0b075ba36f2f2594e59cadd95575eb564d027fd9e" Oct 31 00:30:05.320778 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:05.320754 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv_odh-ai-gateway-infra(b79638dc-69ec-4f25-a8fa-a9baeba3e8fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" Oct 31 00:30:05.322099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:05.322082 4969 scope.go:122] "RemoveContainer" containerID="450260883f33b3224ef94349a0e758b5447079cccd9ab8bcff065aa3e59f56ac" Oct 31 00:30:05.322309 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:05.322293 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp_odh-ai-gateway-infra(17a77832-b33e-4306-8f05-2e034730ebed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" podUID="17a77832-b33e-4306-8f05-2e034730ebed" Oct 31 00:30:09.050776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:09.050741 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb"] Oct 31 00:30:09.100524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:09.100498 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8899f9f-mg2gb"] Oct 31 00:30:09.736241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:09.736211 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39467d37-07ab-4e4f-a519-5ea4b08ced73" path="/var/lib/kubelet/pods/39467d37-07ab-4e4f-a519-5ea4b08ced73/volumes" Oct 31 00:30:10.230885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:10.230858 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:30:14.727553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:14.727526 4969 scope.go:122] "RemoveContainer" containerID="c8e272e9b0b6b5a62cc639e0b075ba36f2f2594e59cadd95575eb564d027fd9e" Oct 31 00:30:15.436018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:15.435987 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" event={"ID":"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc","Type":"ContainerStarted","Data":"fcec4608128ca18f83deac26b2d23875134e33d28a1ebb6442ddade2ae3159cc"} Oct 31 00:30:15.726953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:15.726871 4969 scope.go:122] "RemoveContainer" containerID="0ea7585577effac3956448d49d562aa55a537996f57f0bf41ace99aafabb3d7d" Oct 31 00:30:16.446302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:16.446271 4969 generic.go:356] "Generic (PLEG): container finished" podID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerID="fcec4608128ca18f83deac26b2d23875134e33d28a1ebb6442ddade2ae3159cc" exitCode=60 Oct 31 00:30:16.446729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:16.446351 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" event={"ID":"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc","Type":"ContainerDied","Data":"fcec4608128ca18f83deac26b2d23875134e33d28a1ebb6442ddade2ae3159cc"} Oct 31 00:30:16.446729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:16.446418 4969 scope.go:122] "RemoveContainer" containerID="c8e272e9b0b6b5a62cc639e0b075ba36f2f2594e59cadd95575eb564d027fd9e" Oct 31 00:30:16.446953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:16.446931 4969 scope.go:122] "RemoveContainer" containerID="fcec4608128ca18f83deac26b2d23875134e33d28a1ebb6442ddade2ae3159cc" Oct 31 00:30:16.447317 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:16.447295 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv_odh-ai-gateway-infra(b79638dc-69ec-4f25-a8fa-a9baeba3e8fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" Oct 31 00:30:16.448646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:16.448625 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" event={"ID":"6d208cbe-da72-4e31-b46f-4ae0e3d994b0","Type":"ContainerStarted","Data":"61718a79fa1a92236d681208702080da72d0a26556133b0adfa4cae858d2a252"} Oct 31 00:30:16.573634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:16.573533 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv"] Oct 31 00:30:16.727658 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:16.727633 4969 scope.go:122] "RemoveContainer" containerID="6c254016165a5aba8748c69c27fb5dd49afe46d05885b4edf8fd1b74dfcd7b5e" Oct 31 00:30:16.727885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:16.727796 4969 scope.go:122] "RemoveContainer" containerID="5504334c6d63e13afa35bd4e6cb7a855b578a3ba278f19aca0e9f13bff712db6" Oct 31 00:30:17.460784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.460754 4969 generic.go:356] "Generic (PLEG): container finished" podID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerID="61718a79fa1a92236d681208702080da72d0a26556133b0adfa4cae858d2a252" exitCode=60 Oct 31 00:30:17.461301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.460826 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" event={"ID":"6d208cbe-da72-4e31-b46f-4ae0e3d994b0","Type":"ContainerDied","Data":"61718a79fa1a92236d681208702080da72d0a26556133b0adfa4cae858d2a252"} Oct 31 00:30:17.461301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.460926 4969 scope.go:122] "RemoveContainer" containerID="0ea7585577effac3956448d49d562aa55a537996f57f0bf41ace99aafabb3d7d" Oct 31 00:30:17.461425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.461385 4969 scope.go:122] "RemoveContainer" containerID="61718a79fa1a92236d681208702080da72d0a26556133b0adfa4cae858d2a252" Oct 31 00:30:17.461818 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:17.461787 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw_odh-ai-gateway-infra(6d208cbe-da72-4e31-b46f-4ae0e3d994b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" Oct 31 00:30:17.463018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.462996 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" event={"ID":"f3714929-2dbd-4696-b871-d25966e1af84","Type":"ContainerStarted","Data":"a83d4b8eaf149329fa4e8db8b65e795a9d45d2890c6f417392786a7a1c27cd65"} Oct 31 00:30:17.465056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.465034 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" event={"ID":"ca188ff5-a199-45e7-83cc-78a5d975b140","Type":"ContainerStarted","Data":"663f50650500cec01bd0941a0fb74499d399d8014f4652276bf3c29393466339"} Oct 31 00:30:17.711203 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.711111 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw"] Oct 31 00:30:17.714832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.714807 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:17.738015 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.737992 4969 scope.go:122] "RemoveContainer" containerID="450260883f33b3224ef94349a0e758b5447079cccd9ab8bcff065aa3e59f56ac" Oct 31 00:30:17.811968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.811939 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-maas-api-service-ca\") pod \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\" (UID: \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\") " Oct 31 00:30:17.812282 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.812261 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-maas-api-service-ca" pod "b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" (UID: "b79638dc-69ec-4f25-a8fa-a9baeba3e8fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:17.812466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.812451 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-kube-api-access-4qd5k\" (UniqueName: \"kubernetes.io/projected/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-kube-api-access-4qd5k\") pod \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\" (UID: \"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc\") " Oct 31 00:30:17.813440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.813416 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.819167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.819142 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-kube-api-access-4qd5k" pod "b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" (UID: "b79638dc-69ec-4f25-a8fa-a9baeba3e8fc"). InnerVolumeSpecName "kube-api-access-4qd5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:17.915443 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:17.915414 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qd5k\" (UniqueName: \"kubernetes.io/projected/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc-kube-api-access-4qd5k\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.479570 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.479537 4969 generic.go:356] "Generic (PLEG): container finished" podID="f3714929-2dbd-4696-b871-d25966e1af84" containerID="a83d4b8eaf149329fa4e8db8b65e795a9d45d2890c6f417392786a7a1c27cd65" exitCode=60 Oct 31 00:30:18.480004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.479634 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" event={"ID":"f3714929-2dbd-4696-b871-d25966e1af84","Type":"ContainerDied","Data":"a83d4b8eaf149329fa4e8db8b65e795a9d45d2890c6f417392786a7a1c27cd65"} Oct 31 00:30:18.480004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.479704 4969 scope.go:122] "RemoveContainer" containerID="5504334c6d63e13afa35bd4e6cb7a855b578a3ba278f19aca0e9f13bff712db6" Oct 31 00:30:18.480174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.480155 4969 scope.go:122] "RemoveContainer" containerID="a83d4b8eaf149329fa4e8db8b65e795a9d45d2890c6f417392786a7a1c27cd65" Oct 31 00:30:18.480623 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:18.480557 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s_odh-ai-gateway-infra(f3714929-2dbd-4696-b871-d25966e1af84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" podUID="f3714929-2dbd-4696-b871-d25966e1af84" Oct 31 00:30:18.482019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.481996 4969 generic.go:356] "Generic (PLEG): container finished" podID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerID="663f50650500cec01bd0941a0fb74499d399d8014f4652276bf3c29393466339" exitCode=60 Oct 31 00:30:18.482108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.482067 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" event={"ID":"ca188ff5-a199-45e7-83cc-78a5d975b140","Type":"ContainerDied","Data":"663f50650500cec01bd0941a0fb74499d399d8014f4652276bf3c29393466339"} Oct 31 00:30:18.482506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.482488 4969 scope.go:122] "RemoveContainer" containerID="663f50650500cec01bd0941a0fb74499d399d8014f4652276bf3c29393466339" Oct 31 00:30:18.482917 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:18.482892 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b_odh-ai-gateway-infra(ca188ff5-a199-45e7-83cc-78a5d975b140)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" Oct 31 00:30:18.483937 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.483904 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" event={"ID":"b79638dc-69ec-4f25-a8fa-a9baeba3e8fc","Type":"ContainerDied","Data":"eba428aad3c7544226a2a10e523787d1f5e4aa9ca1178ac6bff75a8c9ba5b402"} Oct 31 00:30:18.484049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.483959 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv" Oct 31 00:30:18.486966 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.486932 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" event={"ID":"17a77832-b33e-4306-8f05-2e034730ebed","Type":"ContainerStarted","Data":"c683c8d2db192bd0896f3c1427415c30fbca58d053fd1b95616731620ee572d4"} Oct 31 00:30:18.497232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.497197 4969 scope.go:122] "RemoveContainer" containerID="6c254016165a5aba8748c69c27fb5dd49afe46d05885b4edf8fd1b74dfcd7b5e" Oct 31 00:30:18.514967 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.514781 4969 scope.go:122] "RemoveContainer" containerID="fcec4608128ca18f83deac26b2d23875134e33d28a1ebb6442ddade2ae3159cc" Oct 31 00:30:18.659207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.659184 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:18.727232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.727203 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-maas-api-service-ca\") pod \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\" (UID: \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\") " Oct 31 00:30:18.727477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.727463 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-kube-api-access-lcx2f\" (UniqueName: \"kubernetes.io/projected/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-kube-api-access-lcx2f\") pod \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\" (UID: \"6d208cbe-da72-4e31-b46f-4ae0e3d994b0\") " Oct 31 00:30:18.727659 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.727631 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-maas-api-service-ca" pod "6d208cbe-da72-4e31-b46f-4ae0e3d994b0" (UID: "6d208cbe-da72-4e31-b46f-4ae0e3d994b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:18.728102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.728083 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.729710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.729683 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-kube-api-access-lcx2f" pod "6d208cbe-da72-4e31-b46f-4ae0e3d994b0" (UID: "6d208cbe-da72-4e31-b46f-4ae0e3d994b0"). InnerVolumeSpecName "kube-api-access-lcx2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:18.822233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.822203 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv"] Oct 31 00:30:18.827562 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.827538 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a4dc2e-29890110-dcztv"] Oct 31 00:30:18.829488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.829464 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcx2f\" (UniqueName: \"kubernetes.io/projected/6d208cbe-da72-4e31-b46f-4ae0e3d994b0-kube-api-access-lcx2f\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.858805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.858776 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s"] Oct 31 00:30:18.976375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:18.976299 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b"] Oct 31 00:30:19.501864 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.501828 4969 generic.go:356] "Generic (PLEG): container finished" podID="17a77832-b33e-4306-8f05-2e034730ebed" containerID="c683c8d2db192bd0896f3c1427415c30fbca58d053fd1b95616731620ee572d4" exitCode=60 Oct 31 00:30:19.502340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.501907 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" event={"ID":"17a77832-b33e-4306-8f05-2e034730ebed","Type":"ContainerDied","Data":"c683c8d2db192bd0896f3c1427415c30fbca58d053fd1b95616731620ee572d4"} Oct 31 00:30:19.502340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.502000 4969 scope.go:122] "RemoveContainer" containerID="450260883f33b3224ef94349a0e758b5447079cccd9ab8bcff065aa3e59f56ac" Oct 31 00:30:19.502446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.502429 4969 scope.go:122] "RemoveContainer" containerID="c683c8d2db192bd0896f3c1427415c30fbca58d053fd1b95616731620ee572d4" Oct 31 00:30:19.502843 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:19.502816 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp_odh-ai-gateway-infra(17a77832-b33e-4306-8f05-2e034730ebed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" podUID="17a77832-b33e-4306-8f05-2e034730ebed" Oct 31 00:30:19.504429 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.504404 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" Oct 31 00:30:19.504429 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.504414 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw" event={"ID":"6d208cbe-da72-4e31-b46f-4ae0e3d994b0","Type":"ContainerDied","Data":"2df8065caa345ee3fd8b251062069f8f3200d6cdac3fee6837b9e41a86f7aab1"} Oct 31 00:30:19.520058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.520035 4969 scope.go:122] "RemoveContainer" containerID="61718a79fa1a92236d681208702080da72d0a26556133b0adfa4cae858d2a252" Oct 31 00:30:19.758020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.757876 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" path="/var/lib/kubelet/pods/b79638dc-69ec-4f25-a8fa-a9baeba3e8fc/volumes" Oct 31 00:30:19.779542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.769836 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:19.780288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.780263 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:19.847063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.847034 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3714929-2dbd-4696-b871-d25966e1af84-kube-api-access-qpdq6\" (UniqueName: \"kubernetes.io/projected/f3714929-2dbd-4696-b871-d25966e1af84-kube-api-access-qpdq6\") pod \"f3714929-2dbd-4696-b871-d25966e1af84\" (UID: \"f3714929-2dbd-4696-b871-d25966e1af84\") " Oct 31 00:30:19.847260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.847099 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca188ff5-a199-45e7-83cc-78a5d975b140-kube-api-access-npzs7\" (UniqueName: \"kubernetes.io/projected/ca188ff5-a199-45e7-83cc-78a5d975b140-kube-api-access-npzs7\") pod \"ca188ff5-a199-45e7-83cc-78a5d975b140\" (UID: \"ca188ff5-a199-45e7-83cc-78a5d975b140\") " Oct 31 00:30:19.847260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.847239 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3714929-2dbd-4696-b871-d25966e1af84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3714929-2dbd-4696-b871-d25966e1af84-maas-api-service-ca\") pod \"f3714929-2dbd-4696-b871-d25966e1af84\" (UID: \"f3714929-2dbd-4696-b871-d25966e1af84\") " Oct 31 00:30:19.847386 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.847353 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca188ff5-a199-45e7-83cc-78a5d975b140-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca188ff5-a199-45e7-83cc-78a5d975b140-maas-api-service-ca\") pod \"ca188ff5-a199-45e7-83cc-78a5d975b140\" (UID: \"ca188ff5-a199-45e7-83cc-78a5d975b140\") " Oct 31 00:30:19.847734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.847704 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3714929-2dbd-4696-b871-d25966e1af84-maas-api-service-ca" pod "f3714929-2dbd-4696-b871-d25966e1af84" (UID: "f3714929-2dbd-4696-b871-d25966e1af84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:19.848079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.847732 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca188ff5-a199-45e7-83cc-78a5d975b140-maas-api-service-ca" pod "ca188ff5-a199-45e7-83cc-78a5d975b140" (UID: "ca188ff5-a199-45e7-83cc-78a5d975b140"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:19.848152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.848118 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3714929-2dbd-4696-b871-d25966e1af84-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.848152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.848134 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca188ff5-a199-45e7-83cc-78a5d975b140-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.850218 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.850192 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca188ff5-a199-45e7-83cc-78a5d975b140-kube-api-access-npzs7" pod "ca188ff5-a199-45e7-83cc-78a5d975b140" (UID: "ca188ff5-a199-45e7-83cc-78a5d975b140"). InnerVolumeSpecName "kube-api-access-npzs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:19.850491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.850463 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3714929-2dbd-4696-b871-d25966e1af84-kube-api-access-qpdq6" pod "f3714929-2dbd-4696-b871-d25966e1af84" (UID: "f3714929-2dbd-4696-b871-d25966e1af84"). InnerVolumeSpecName "kube-api-access-qpdq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:19.949693 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.949661 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpdq6\" (UniqueName: \"kubernetes.io/projected/f3714929-2dbd-4696-b871-d25966e1af84-kube-api-access-qpdq6\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.949693 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.949691 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npzs7\" (UniqueName: \"kubernetes.io/projected/ca188ff5-a199-45e7-83cc-78a5d975b140-kube-api-access-npzs7\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.980603 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.980559 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw"] Oct 31 00:30:19.985931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:19.985902 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-835921-29890110-ghtzw"] Oct 31 00:30:20.089045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.088967 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k"] Oct 31 00:30:20.103652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.103620 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp"] Oct 31 00:30:20.138217 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.138182 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45ad2e0-bng2k"] Oct 31 00:30:20.522191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.522164 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" Oct 31 00:30:20.522689 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.522164 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s" event={"ID":"f3714929-2dbd-4696-b871-d25966e1af84","Type":"ContainerDied","Data":"11d38c6c6b8ea9c8b671c3d5b24df32bfaead86376b3423353e3a86856eb8162"} Oct 31 00:30:20.522689 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.522313 4969 scope.go:122] "RemoveContainer" containerID="a83d4b8eaf149329fa4e8db8b65e795a9d45d2890c6f417392786a7a1c27cd65" Oct 31 00:30:20.524091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.523828 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" Oct 31 00:30:20.524091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.523885 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b" event={"ID":"ca188ff5-a199-45e7-83cc-78a5d975b140","Type":"ContainerDied","Data":"7a1ff023f17268082916a37504d4b944405891e145610024177d2fdcccf3bde2"} Oct 31 00:30:20.537314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.537289 4969 scope.go:122] "RemoveContainer" containerID="663f50650500cec01bd0941a0fb74499d399d8014f4652276bf3c29393466339" Oct 31 00:30:20.691302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.691268 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s"] Oct 31 00:30:20.697033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.697004 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890110-4hp6s"] Oct 31 00:30:20.723217 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.723193 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:20.871806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.871776 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a77832-b33e-4306-8f05-2e034730ebed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a77832-b33e-4306-8f05-2e034730ebed-maas-api-service-ca\") pod \"17a77832-b33e-4306-8f05-2e034730ebed\" (UID: \"17a77832-b33e-4306-8f05-2e034730ebed\") " Oct 31 00:30:20.872060 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.872040 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a77832-b33e-4306-8f05-2e034730ebed-kube-api-access-pv485\" (UniqueName: \"kubernetes.io/projected/17a77832-b33e-4306-8f05-2e034730ebed-kube-api-access-pv485\") pod \"17a77832-b33e-4306-8f05-2e034730ebed\" (UID: \"17a77832-b33e-4306-8f05-2e034730ebed\") " Oct 31 00:30:20.872174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.872148 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a77832-b33e-4306-8f05-2e034730ebed-maas-api-service-ca" pod "17a77832-b33e-4306-8f05-2e034730ebed" (UID: "17a77832-b33e-4306-8f05-2e034730ebed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:20.872507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.872490 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a77832-b33e-4306-8f05-2e034730ebed-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:20.874347 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.874323 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a77832-b33e-4306-8f05-2e034730ebed-kube-api-access-pv485" pod "17a77832-b33e-4306-8f05-2e034730ebed" (UID: "17a77832-b33e-4306-8f05-2e034730ebed"). InnerVolumeSpecName "kube-api-access-pv485". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:20.915223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.915146 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b"] Oct 31 00:30:20.920233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.920210 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10a7aa-29890110-sjc4b"] Oct 31 00:30:20.974190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:20.974163 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv485\" (UniqueName: \"kubernetes.io/projected/17a77832-b33e-4306-8f05-2e034730ebed-kube-api-access-pv485\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:21.462165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.462112 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7504-d00901"] Oct 31 00:30:21.463635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463611 4969 cpu_manager.go:414] "containerMap: removing container" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerName="cleanup" Oct 31 00:30:21.463723 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463639 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerName="cleanup" Oct 31 00:30:21.463723 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463680 4969 cpu_manager.go:414] "containerMap: removing container" podUID="f3714929-2dbd-4696-b871-d25966e1af84" containerName="cleanup" Oct 31 00:30:21.463723 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463687 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3714929-2dbd-4696-b871-d25966e1af84" containerName="cleanup" Oct 31 00:30:21.463723 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463706 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerName="cleanup" Oct 31 00:30:21.463723 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463714 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463727 4969 cpu_manager.go:414] "containerMap: removing container" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463734 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463751 4969 cpu_manager.go:414] "containerMap: removing container" podUID="17a77832-b33e-4306-8f05-2e034730ebed" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463758 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a77832-b33e-4306-8f05-2e034730ebed" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463777 4969 cpu_manager.go:414] "containerMap: removing container" podUID="f3714929-2dbd-4696-b871-d25966e1af84" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463783 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3714929-2dbd-4696-b871-d25966e1af84" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463798 4969 cpu_manager.go:414] "containerMap: removing container" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463806 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463839 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463845 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463879 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463886 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463902 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerName="cleanup" Oct 31 00:30:21.463906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463908 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463944 4969 cpu_manager.go:414] "containerMap: removing container" podUID="17a77832-b33e-4306-8f05-2e034730ebed" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463951 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a77832-b33e-4306-8f05-2e034730ebed" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463978 4969 cpu_manager.go:414] "containerMap: removing container" podUID="17a77832-b33e-4306-8f05-2e034730ebed" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.463984 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a77832-b33e-4306-8f05-2e034730ebed" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464292 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464310 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3714929-2dbd-4696-b871-d25966e1af84" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464328 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464339 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464355 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a77832-b33e-4306-8f05-2e034730ebed" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464363 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3714929-2dbd-4696-b871-d25966e1af84" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464383 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464399 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464406 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464413 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464431 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerName="cleanup" Oct 31 00:30:21.464518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.464440 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a77832-b33e-4306-8f05-2e034730ebed" containerName="cleanup" Oct 31 00:30:21.469328 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.469305 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:21.513115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.513088 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-d00901"] Oct 31 00:30:21.522087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.522061 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9d9vc\"" Oct 31 00:30:21.542547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.542510 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" event={"ID":"17a77832-b33e-4306-8f05-2e034730ebed","Type":"ContainerDied","Data":"cfea1f950e537fdcc88503fd188b68d7776e92a54b7ea9bcff8c719ca79c189a"} Oct 31 00:30:21.543026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.542549 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp" Oct 31 00:30:21.543026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.542583 4969 scope.go:122] "RemoveContainer" containerID="c683c8d2db192bd0896f3c1427415c30fbca58d053fd1b95616731620ee572d4" Oct 31 00:30:21.582673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.582638 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2rh\" (UniqueName: \"kubernetes.io/projected/095a4ece-88c1-441b-b516-c5891038340e-kube-api-access-br2rh\") pod \"test-curl-7504-d00901\" (UID: \"095a4ece-88c1-441b-b516-c5891038340e\") " pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:21.684557 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.684524 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2rh\" (UniqueName: \"kubernetes.io/projected/095a4ece-88c1-441b-b516-c5891038340e-kube-api-access-br2rh\") pod \"test-curl-7504-d00901\" (UID: \"095a4ece-88c1-441b-b516-c5891038340e\") " pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:21.697078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.697048 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp"] Oct 31 00:30:21.701725 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.701698 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890110-p8xhp"] Oct 31 00:30:21.737128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.737061 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a77832-b33e-4306-8f05-2e034730ebed" path="/var/lib/kubelet/pods/17a77832-b33e-4306-8f05-2e034730ebed/volumes" Oct 31 00:30:21.737604 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.737577 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3add4aa1-b87d-426a-9dce-c24a28f0b608" path="/var/lib/kubelet/pods/3add4aa1-b87d-426a-9dce-c24a28f0b608/volumes" Oct 31 00:30:21.737976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.737963 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" path="/var/lib/kubelet/pods/6d208cbe-da72-4e31-b46f-4ae0e3d994b0/volumes" Oct 31 00:30:21.738692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.738677 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" path="/var/lib/kubelet/pods/ca188ff5-a199-45e7-83cc-78a5d975b140/volumes" Oct 31 00:30:21.739064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.739050 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3714929-2dbd-4696-b871-d25966e1af84" path="/var/lib/kubelet/pods/f3714929-2dbd-4696-b871-d25966e1af84/volumes" Oct 31 00:30:21.745672 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.745651 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2rh\" (UniqueName: \"kubernetes.io/projected/095a4ece-88c1-441b-b516-c5891038340e-kube-api-access-br2rh\") pod \"test-curl-7504-d00901\" (UID: \"095a4ece-88c1-441b-b516-c5891038340e\") " pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:21.784188 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:21.784160 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:22.170384 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:22.170352 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod095a4ece_88c1_441b_b516_c5891038340e.slice/crio-559e71c77b58c1384be24385a9cf3b822feeac6ee4a8e2e4cd1d0a776f0d2e0b WatchSource:0}: Error finding container 559e71c77b58c1384be24385a9cf3b822feeac6ee4a8e2e4cd1d0a776f0d2e0b: Status 404 returned error can't find the container with id 559e71c77b58c1384be24385a9cf3b822feeac6ee4a8e2e4cd1d0a776f0d2e0b Oct 31 00:30:22.171875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:22.171849 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-d00901"] Oct 31 00:30:22.172427 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:22.172410 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:30:22.556880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:22.556833 4969 generic.go:356] "Generic (PLEG): container finished" podID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerID="1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82" exitCode=0 Oct 31 00:30:22.557309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:22.556908 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" event={"ID":"05e6a1ab-f335-42fc-a645-b18f8d554ead","Type":"ContainerDied","Data":"1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82"} Oct 31 00:30:22.557367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:22.557338 4969 scope.go:122] "RemoveContainer" containerID="1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82" Oct 31 00:30:22.559011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:22.558978 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:22.559145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:22.559077 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:22.559145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:22.559113 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:22.559688 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:22.559539 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-d00901" event={"ID":"095a4ece-88c1-441b-b516-c5891038340e","Type":"ContainerStarted","Data":"559e71c77b58c1384be24385a9cf3b822feeac6ee4a8e2e4cd1d0a776f0d2e0b"} Oct 31 00:30:23.570671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:23.570641 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" event={"ID":"05e6a1ab-f335-42fc-a645-b18f8d554ead","Type":"ContainerStarted","Data":"3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b"} Oct 31 00:30:24.074833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:24.074796 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:30:24.075098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:24.075065 4969 patch_prober.go:29] interesting pod/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:30:24.075226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:24.075118 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 00:30:25.075734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:25.075695 4969 patch_prober.go:29] interesting pod/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:30:25.076220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:25.075764 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 00:30:25.616878 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:25.616835 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-d00901" event={"ID":"095a4ece-88c1-441b-b516-c5891038340e","Type":"ContainerStarted","Data":"5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5"} Oct 31 00:30:25.734424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:25.734374 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7504-d00901" podStartSLOduration=1.582740606 podStartE2EDuration="4.734360071s" totalImagesPullingTime="3.151619465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:25.729869054 +0000 UTC m=+1658.205227478" watchObservedRunningTime="2026-10-31 00:30:25.734360071 +0000 UTC m=+1658.209718494" Oct 31 00:30:26.075928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:26.075833 4969 patch_prober.go:29] interesting pod/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:30:26.075928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:26.075906 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 00:30:27.074999 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:27.074962 4969 patch_prober.go:29] interesting pod/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:30:27.075173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:27.075012 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 00:30:27.337060 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:27.336966 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7504-d00901"] Oct 31 00:30:27.342351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:27.342323 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7504-d00901"] Oct 31 00:30:27.633716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:27.633680 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7504-d00901" podUID="095a4ece-88c1-441b-b516-c5891038340e" containerName="test-curl-7504-d00901" containerID="cri-o://5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5" gracePeriod=2 Oct 31 00:30:27.699616 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:27.692823 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7504-d00901\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="095a4ece-88c1-441b-b516-c5891038340e" pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:27.801617 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:27.800184 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7504-d00901\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="095a4ece-88c1-441b-b516-c5891038340e" pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:28.075334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.075234 4969 patch_prober.go:29] interesting pod/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:30:28.075334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.075311 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 00:30:28.404678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.404635 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs"] Oct 31 00:30:28.405291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405274 4969 cpu_manager.go:414] "containerMap: removing container" podUID="f3714929-2dbd-4696-b871-d25966e1af84" containerName="cleanup" Oct 31 00:30:28.405291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405290 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3714929-2dbd-4696-b871-d25966e1af84" containerName="cleanup" Oct 31 00:30:28.405430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405311 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerName="cleanup" Oct 31 00:30:28.405430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405316 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca188ff5-a199-45e7-83cc-78a5d975b140" containerName="cleanup" Oct 31 00:30:28.405430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405329 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerName="cleanup" Oct 31 00:30:28.405430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405334 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79638dc-69ec-4f25-a8fa-a9baeba3e8fc" containerName="cleanup" Oct 31 00:30:28.405430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405366 4969 cpu_manager.go:414] "containerMap: removing container" podUID="095a4ece-88c1-441b-b516-c5891038340e" containerName="test-curl-7504-d00901" Oct 31 00:30:28.405430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405371 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095a4ece-88c1-441b-b516-c5891038340e" containerName="test-curl-7504-d00901" Oct 31 00:30:28.405717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405467 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="095a4ece-88c1-441b-b516-c5891038340e" containerName="test-curl-7504-d00901" Oct 31 00:30:28.405717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405475 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d208cbe-da72-4e31-b46f-4ae0e3d994b0" containerName="cleanup" Oct 31 00:30:28.405717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405484 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a77832-b33e-4306-8f05-2e034730ebed" containerName="cleanup" Oct 31 00:30:28.405717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.405489 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3714929-2dbd-4696-b871-d25966e1af84" containerName="cleanup" Oct 31 00:30:28.409304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.409289 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.447508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.447484 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs"] Oct 31 00:30:28.506627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.505245 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.506627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.505358 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9chrg\" (UniqueName: \"kubernetes.io/projected/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-kube-api-access-9chrg\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.506627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.505408 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:30:28.506627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.505359 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:30:28.506627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.505671 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.606826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.606780 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.607033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.606863 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.607033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.606943 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9chrg\" (UniqueName: \"kubernetes.io/projected/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-kube-api-access-9chrg\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.607603 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.607500 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.609288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.609264 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.667647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.667569 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chrg\" (UniqueName: \"kubernetes.io/projected/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-kube-api-access-9chrg\") pod \"maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:28.732073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:28.732041 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:29.075351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:29.075320 4969 patch_prober.go:29] interesting pod/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 00:30:29.075536 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:29.075376 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 00:30:29.296156 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:29.296130 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdf5a3d_bf3e_49fc_a1c0_b0be0ee8e0c1.slice/crio-b287ba4e28a3019046356dd77b10dbd4408bba1ae0dd668824bf97c21ea4145c WatchSource:0}: Error finding container b287ba4e28a3019046356dd77b10dbd4408bba1ae0dd668824bf97c21ea4145c: Status 404 returned error can't find the container with id b287ba4e28a3019046356dd77b10dbd4408bba1ae0dd668824bf97c21ea4145c Oct 31 00:30:29.297183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:29.297162 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs"] Oct 31 00:30:29.653317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:29.653281 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" event={"ID":"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1","Type":"ContainerStarted","Data":"a268c087003b5f15f3e4744e37f97c8e2e3fe5e698400801d2c310529f3b8280"} Oct 31 00:30:29.653706 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:29.653347 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" event={"ID":"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1","Type":"ContainerStarted","Data":"b287ba4e28a3019046356dd77b10dbd4408bba1ae0dd668824bf97c21ea4145c"} Oct 31 00:30:29.782192 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:29.782138 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" podStartSLOduration=1.782125675 podStartE2EDuration="1.782125675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:29.780362444 +0000 UTC m=+1662.255720868" watchObservedRunningTime="2026-10-31 00:30:29.782125675 +0000 UTC m=+1662.257484096" Oct 31 00:30:29.897299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:29.897271 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:30.030271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.030230 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095a4ece-88c1-441b-b516-c5891038340e-kube-api-access-br2rh\" (UniqueName: \"kubernetes.io/projected/095a4ece-88c1-441b-b516-c5891038340e-kube-api-access-br2rh\") pod \"095a4ece-88c1-441b-b516-c5891038340e\" (UID: \"095a4ece-88c1-441b-b516-c5891038340e\") " Oct 31 00:30:30.032521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.032496 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095a4ece-88c1-441b-b516-c5891038340e-kube-api-access-br2rh" pod "095a4ece-88c1-441b-b516-c5891038340e" (UID: "095a4ece-88c1-441b-b516-c5891038340e"). InnerVolumeSpecName "kube-api-access-br2rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:30.080746 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.080722 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:30:30.081249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.081224 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:30:30.081893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.081876 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:30:30.132142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.132117 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br2rh\" (UniqueName: \"kubernetes.io/projected/095a4ece-88c1-441b-b516-c5891038340e-kube-api-access-br2rh\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.662888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.662857 4969 generic.go:356] "Generic (PLEG): container finished" podID="095a4ece-88c1-441b-b516-c5891038340e" containerID="5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5" exitCode=137 Oct 31 00:30:30.663341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.662922 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:30.663341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.662961 4969 scope.go:122] "RemoveContainer" containerID="5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5" Oct 31 00:30:30.665152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.665125 4969 generic.go:356] "Generic (PLEG): container finished" podID="bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1" containerID="a268c087003b5f15f3e4744e37f97c8e2e3fe5e698400801d2c310529f3b8280" exitCode=0 Oct 31 00:30:30.665288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.665208 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" event={"ID":"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1","Type":"ContainerDied","Data":"a268c087003b5f15f3e4744e37f97c8e2e3fe5e698400801d2c310529f3b8280"} Oct 31 00:30:30.678768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.678748 4969 scope.go:122] "RemoveContainer" containerID="5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5" Oct 31 00:30:30.679075 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:30.679051 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5\": container with ID starting with 5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5 not found: ID does not exist" containerID="5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5" Oct 31 00:30:30.679175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:30.679080 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5"} err="failed to get container status \"5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5\": rpc error: code = NotFound desc = could not find container \"5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5\": container with ID starting with 5191037671e6a8eee44b34f00a26719a1d593d761f2df24ff9bc66dcbb9278b5 not found: ID does not exist" Oct 31 00:30:30.830902 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:30.830860 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7504-d00901\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="095a4ece-88c1-441b-b516-c5891038340e" pod="opendatahub/test-curl-7504-d00901" Oct 31 00:30:31.738715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:31.738191 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095a4ece-88c1-441b-b516-c5891038340e" path="/var/lib/kubelet/pods/095a4ece-88c1-441b-b516-c5891038340e/volumes" Oct 31 00:30:31.842338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:31.842307 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:31.962388 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:31.962309 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-openshift-ca\") pod \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " Oct 31 00:30:31.962388 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:31.962368 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-kube-api-access-9chrg\" (UniqueName: \"kubernetes.io/projected/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-kube-api-access-9chrg\") pod \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " Oct 31 00:30:31.962661 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:31.962641 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-xks-ca\") pod \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\" (UID: \"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1\") " Oct 31 00:30:31.962754 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:31.962730 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-openshift-ca" pod "bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1" (UID: "bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:31.963206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:31.963184 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:31.964762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:31.964732 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-kube-api-access-9chrg" pod "bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1" (UID: "bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1"). InnerVolumeSpecName "kube-api-access-9chrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:31.964852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:31.964764 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-xks-ca" pod "bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1" (UID: "bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:32.051531 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.051501 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr"] Oct 31 00:30:32.056345 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.056314 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b475ef40-fstxr"] Oct 31 00:30:32.064731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.064707 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9chrg\" (UniqueName: \"kubernetes.io/projected/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-kube-api-access-9chrg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:32.064731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.064731 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:30:32.489077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.489049 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf"] Oct 31 00:30:32.489739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.489716 4969 cpu_manager.go:414] "containerMap: removing container" podUID="bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1" containerName="revoke-keys" Oct 31 00:30:32.489739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.489732 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1" containerName="revoke-keys" Oct 31 00:30:32.489891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.489841 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1" containerName="revoke-keys" Oct 31 00:30:32.494717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.494698 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.538957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.538930 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7"] Oct 31 00:30:32.543274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.543251 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.572010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.571980 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzlb\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-kube-api-access-6hzlb\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.572119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.572100 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.572173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.572161 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.572212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.572185 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.572254 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.572209 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.572338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.572325 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.572373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.572358 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.572438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.572425 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.572479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.572468 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.597135 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.597087 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf"] Oct 31 00:30:32.607212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.607183 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7"] Oct 31 00:30:32.673451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673416 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.673657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673457 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.673657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673568 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.673657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673636 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.673852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673677 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v25tj\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-kube-api-access-v25tj\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.673852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673746 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.673852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673818 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzlb\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-kube-api-access-6hzlb\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.674012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673859 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.674012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673863 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.674012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.673932 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.674170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674066 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.674170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674150 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decc71b4-6554-457b-86c9-969fe79ba955-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.674279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674209 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.674346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674281 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.674346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674317 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.674447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674360 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decc71b4-6554-457b-86c9-969fe79ba955-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.674500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674460 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.674500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674490 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.674649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674518 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.674649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674553 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.674768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674554 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.674958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.674919 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.675201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.675166 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.678525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.677059 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.678525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.677477 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.689459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.689434 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" event={"ID":"bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1","Type":"ContainerDied","Data":"b287ba4e28a3019046356dd77b10dbd4408bba1ae0dd668824bf97c21ea4145c"} Oct 31 00:30:32.689627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.689475 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs" Oct 31 00:30:32.689627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.689479 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b287ba4e28a3019046356dd77b10dbd4408bba1ae0dd668824bf97c21ea4145c" Oct 31 00:30:32.729659 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:30:32.729623 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:30:32.737024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.737001 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzlb\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-kube-api-access-6hzlb\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.737146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.737035 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.776073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776009 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776080 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decc71b4-6554-457b-86c9-969fe79ba955-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776142 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776193 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776232 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decc71b4-6554-457b-86c9-969fe79ba955-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776293 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776430 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776461 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v25tj\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-kube-api-access-v25tj\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776471 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776721 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776792 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.776925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.776827 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.777074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.777011 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.777195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.777173 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decc71b4-6554-457b-86c9-969fe79ba955-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.779093 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.779071 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.779380 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.779361 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decc71b4-6554-457b-86c9-969fe79ba955-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.809567 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.809539 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:32.837825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.837773 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.838457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.838389 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v25tj\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-kube-api-access-v25tj\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:32.856538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:32.856501 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:33.259719 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:33.259688 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf6cadb_2c52_4481_a3a9_f20c6228f48a.slice/crio-6a51776c049e736f31ab000c40e954875469b6c3e622f58464c844c2fc9bbe13 WatchSource:0}: Error finding container 6a51776c049e736f31ab000c40e954875469b6c3e622f58464c844c2fc9bbe13: Status 404 returned error can't find the container with id 6a51776c049e736f31ab000c40e954875469b6c3e622f58464c844c2fc9bbe13 Oct 31 00:30:33.260332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.260308 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf"] Oct 31 00:30:33.262980 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.262946 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:33.263096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.263030 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:33.263096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.263076 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:33.312118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.312087 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7"] Oct 31 00:30:33.331908 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:33.324330 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecc71b4_6554_457b_86c9_969fe79ba955.slice/crio-64ab076fe64afb6f5d2979b07bdcadc71578e656be6ba2189ee4e8ef4dec2373 WatchSource:0}: Error finding container 64ab076fe64afb6f5d2979b07bdcadc71578e656be6ba2189ee4e8ef4dec2373: Status 404 returned error can't find the container with id 64ab076fe64afb6f5d2979b07bdcadc71578e656be6ba2189ee4e8ef4dec2373 Oct 31 00:30:33.331908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.327757 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:33.331908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.327890 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:33.331908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.327959 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:33.702345 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.702307 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" event={"ID":"edf6cadb-2c52-4481-a3a9-f20c6228f48a","Type":"ContainerStarted","Data":"7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb"} Oct 31 00:30:33.702522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.702378 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" event={"ID":"edf6cadb-2c52-4481-a3a9-f20c6228f48a","Type":"ContainerStarted","Data":"6a51776c049e736f31ab000c40e954875469b6c3e622f58464c844c2fc9bbe13"} Oct 31 00:30:33.704780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.704744 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" event={"ID":"decc71b4-6554-457b-86c9-969fe79ba955","Type":"ContainerStarted","Data":"9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab"} Oct 31 00:30:33.705112 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.704790 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" event={"ID":"decc71b4-6554-457b-86c9-969fe79ba955","Type":"ContainerStarted","Data":"64ab076fe64afb6f5d2979b07bdcadc71578e656be6ba2189ee4e8ef4dec2373"} Oct 31 00:30:33.739762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.739731 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4172a010-d35c-43b6-9234-b867c7227f25" path="/var/lib/kubelet/pods/4172a010-d35c-43b6-9234-b867c7227f25/volumes" Oct 31 00:30:33.810244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.810213 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:33.856704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.856673 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:33.973757 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.973652 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" podStartSLOduration=1.973638533 podStartE2EDuration="1.973638533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:33.833503806 +0000 UTC m=+1666.308862229" watchObservedRunningTime="2026-10-31 00:30:33.973638533 +0000 UTC m=+1666.448996956" Oct 31 00:30:33.996618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:33.995769 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" podStartSLOduration=1.995749107 podStartE2EDuration="1.995749107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:33.969731327 +0000 UTC m=+1666.445089752" watchObservedRunningTime="2026-10-31 00:30:33.995749107 +0000 UTC m=+1666.471107530" Oct 31 00:30:34.816620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:34.812993 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:30:34.816620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:34.813058 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 00:30:34.859135 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:34.859099 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:30:34.859305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:34.859162 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 00:30:35.815614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:35.815058 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:30:35.815614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:35.815135 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 00:30:35.857928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:35.857886 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:30:35.858324 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:35.857952 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 00:30:36.050617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.050091 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp"] Oct 31 00:30:36.117702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.115906 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.162960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.161772 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b"] Oct 31 00:30:36.168126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.168069 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e2cf70-x4l7b"] Oct 31 00:30:36.170006 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.169967 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3e7872-openshift-default-dockercfg-ldnlm\"" Oct 31 00:30:36.175539 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.175513 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp"] Oct 31 00:30:36.243617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.232791 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-credential-socket\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.243617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.232891 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-socket\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.245109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.245063 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp75h\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-kube-api-access-jp75h\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.252833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.252800 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istiod-ca-cert\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.263296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.263265 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-envoy\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.271151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.271121 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-podinfo\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.274738 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.274704 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-token\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.274938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.274819 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-certs\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.274938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.274925 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-data\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.376191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.376105 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp75h\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-kube-api-access-jp75h\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.376354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.376189 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istiod-ca-cert\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.376354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.376229 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-envoy\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.376354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.376261 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-podinfo\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.376520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.376353 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-token\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.376520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.376393 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-certs\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.376520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.376473 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-data\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.376520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.376504 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-credential-socket\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.380863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.376544 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-socket\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.404618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.404242 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istiod-ca-cert\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.408614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.408370 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-credential-socket\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.408614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.408543 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-certs\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.408821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.408694 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-socket\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.408821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.408786 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-data\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.432946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.432444 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-envoy\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.435210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.435182 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-podinfo\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.460541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.460515 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-token\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.460825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.460802 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp75h\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-kube-api-access-jp75h\") pod \"e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.497492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.497455 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:36.834637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.832674 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:30:36.834637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.832733 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 00:30:36.857350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.857319 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:30:36.857541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:36.857519 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 00:30:37.066941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.066911 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp"] Oct 31 00:30:37.067780 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:37.067730 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a5d113_45e6_4ff3_9ac6_a2c4a3d1a611.slice/crio-9d77e3fd57dbcb64dd7982f4bfe9e13f1f2ca413e5a98687ef2bdfd76b18ee05 WatchSource:0}: Error finding container 9d77e3fd57dbcb64dd7982f4bfe9e13f1f2ca413e5a98687ef2bdfd76b18ee05: Status 404 returned error can't find the container with id 9d77e3fd57dbcb64dd7982f4bfe9e13f1f2ca413e5a98687ef2bdfd76b18ee05 Oct 31 00:30:37.072458 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.072415 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:37.072603 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.072562 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:37.072726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.072698 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:37.757760 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.757718 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f6f609-2edb-40cc-bdf9-2d814ff68891" path="/var/lib/kubelet/pods/93f6f609-2edb-40cc-bdf9-2d814ff68891/volumes" Oct 31 00:30:37.798621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.796517 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" event={"ID":"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611","Type":"ContainerStarted","Data":"501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002"} Oct 31 00:30:37.798621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.796610 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" event={"ID":"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611","Type":"ContainerStarted","Data":"9d77e3fd57dbcb64dd7982f4bfe9e13f1f2ca413e5a98687ef2bdfd76b18ee05"} Oct 31 00:30:37.835616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.831822 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:30:37.835616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.831882 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 00:30:37.872618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.871742 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:30:37.872618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:37.871800 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 00:30:38.195902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:38.172171 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" podStartSLOduration=3.172150258 podStartE2EDuration="3.172150258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:37.951306739 +0000 UTC m=+1670.426665168" watchObservedRunningTime="2026-10-31 00:30:38.172150258 +0000 UTC m=+1670.647508681" Oct 31 00:30:38.501668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:38.500722 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:38.514935 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:38.514907 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:38.824833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:38.824740 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:30:38.825083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:38.825054 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 00:30:38.837770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:38.837739 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:38.839634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:38.839563 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:30:38.857814 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:38.857783 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:30:38.858062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:38.858036 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 00:30:39.809993 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:39.809956 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 31 00:30:39.810370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:39.810023 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 31 00:30:39.857766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:39.857729 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:30:39.857964 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:39.857801 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 00:30:41.056220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:41.056188 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:41.057247 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:41.057222 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:41.058143 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:41.058121 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:30:41.130163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:41.130119 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:41.130911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:41.130883 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:41.132243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:41.132221 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:30:45.857824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:45.857788 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g"] Oct 31 00:30:45.862350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:45.862331 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:45.918949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:45.918914 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3e7872\"" Oct 31 00:30:45.919318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:45.919300 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3e7872-dockercfg-pg6mk\"" Oct 31 00:30:46.006234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.006203 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g"] Oct 31 00:30:46.036656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.031574 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqvgf\" (UniqueName: \"kubernetes.io/projected/d75a9303-d87f-4ba8-81e1-2235dcf7685b-kube-api-access-cqvgf\") pod \"payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g\" (UID: \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:46.041521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.041493 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75a9303-d87f-4ba8-81e1-2235dcf7685b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g\" (UID: \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:46.086800 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.086749 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f"] Oct 31 00:30:46.093274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.093197 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:46.144241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.144202 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqvgf\" (UniqueName: \"kubernetes.io/projected/d75a9303-d87f-4ba8-81e1-2235dcf7685b-kube-api-access-cqvgf\") pod \"payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g\" (UID: \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:46.144414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.144314 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75a9303-d87f-4ba8-81e1-2235dcf7685b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g\" (UID: \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:46.145058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.145035 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75a9303-d87f-4ba8-81e1-2235dcf7685b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g\" (UID: \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:46.155391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.155363 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f"] Oct 31 00:30:46.205120 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.205091 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqvgf\" (UniqueName: \"kubernetes.io/projected/d75a9303-d87f-4ba8-81e1-2235dcf7685b-kube-api-access-cqvgf\") pod \"payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g\" (UID: \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:46.245460 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.245424 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dmd\" (UniqueName: \"kubernetes.io/projected/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-kube-api-access-75dmd\") pod \"payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f\" (UID: \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:46.245686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.245541 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f\" (UID: \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:46.347324 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.347290 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dmd\" (UniqueName: \"kubernetes.io/projected/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-kube-api-access-75dmd\") pod \"payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f\" (UID: \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:46.347470 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.347448 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f\" (UID: \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:46.348163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.348144 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f\" (UID: \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:46.408154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.408083 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dmd\" (UniqueName: \"kubernetes.io/projected/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-kube-api-access-75dmd\") pod \"payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f\" (UID: \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:46.413003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.412983 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:46.476854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.476810 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:46.800034 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:46.800005 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc2ac6f_fab4_4d04_bd1e_fa768c6c91ec.slice/crio-93777f5a2fbf49f10b97b9b713a10c8776582115118dbf2fc15b10daf10b92a6 WatchSource:0}: Error finding container 93777f5a2fbf49f10b97b9b713a10c8776582115118dbf2fc15b10daf10b92a6: Status 404 returned error can't find the container with id 93777f5a2fbf49f10b97b9b713a10c8776582115118dbf2fc15b10daf10b92a6 Oct 31 00:30:46.801161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.801134 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f"] Oct 31 00:30:46.905187 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:46.905159 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75a9303_d87f_4ba8_81e1_2235dcf7685b.slice/crio-aa8aec9c2fe73b077cb9ca785432ae89a10a8fdda9e426adb233ce5299b6532f WatchSource:0}: Error finding container aa8aec9c2fe73b077cb9ca785432ae89a10a8fdda9e426adb233ce5299b6532f: Status 404 returned error can't find the container with id aa8aec9c2fe73b077cb9ca785432ae89a10a8fdda9e426adb233ce5299b6532f Oct 31 00:30:46.906675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:46.906650 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g"] Oct 31 00:30:47.017061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.017025 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" event={"ID":"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec","Type":"ContainerStarted","Data":"7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe"} Oct 31 00:30:47.017205 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.017082 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" event={"ID":"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec","Type":"ContainerStarted","Data":"93777f5a2fbf49f10b97b9b713a10c8776582115118dbf2fc15b10daf10b92a6"} Oct 31 00:30:47.017205 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.017122 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:47.018889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.018865 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" event={"ID":"d75a9303-d87f-4ba8-81e1-2235dcf7685b","Type":"ContainerStarted","Data":"680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952"} Oct 31 00:30:47.019013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.018901 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" event={"ID":"d75a9303-d87f-4ba8-81e1-2235dcf7685b","Type":"ContainerStarted","Data":"aa8aec9c2fe73b077cb9ca785432ae89a10a8fdda9e426adb233ce5299b6532f"} Oct 31 00:30:47.019013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.018959 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:47.143710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.143654 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" podStartSLOduration=1.143640863 podStartE2EDuration="1.143640863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:47.138486009 +0000 UTC m=+1679.613844435" watchObservedRunningTime="2026-10-31 00:30:47.143640863 +0000 UTC m=+1679.618999286" Oct 31 00:30:47.268056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.267992 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" podStartSLOduration=2.267967577 podStartE2EDuration="2.267967577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:47.256448014 +0000 UTC m=+1679.731806438" watchObservedRunningTime="2026-10-31 00:30:47.267967577 +0000 UTC m=+1679.743326002" Oct 31 00:30:47.793609 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.793566 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76"] Oct 31 00:30:47.798389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.798372 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:47.856617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.855996 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bba5ed-openshift-default-dockercfg-sv8cl\"" Oct 31 00:30:47.978091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.978065 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76"] Oct 31 00:30:47.991755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.991721 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-token\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:47.992105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.992078 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-podinfo\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:47.992327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.992310 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-credential-socket\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:47.992631 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.992577 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8ad439-95c6-4274-8577-4719de10fb3f-istiod-ca-cert\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:47.992867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.992838 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7nf5\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-kube-api-access-m7nf5\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:47.993059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.993042 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-data\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:47.993300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.993282 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-socket\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:47.993477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.993460 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-envoy\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:47.993707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:47.993688 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-certs\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.050981 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.050896 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc"] Oct 31 00:30:48.056820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.056673 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5bd7ac-zncsc"] Oct 31 00:30:48.095260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095213 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-credential-socket\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.095471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095286 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8ad439-95c6-4274-8577-4719de10fb3f-istiod-ca-cert\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.095471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095401 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7nf5\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-kube-api-access-m7nf5\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.095471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095460 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-data\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095565 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-socket\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095618 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-envoy\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095721 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-certs\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095799 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-token\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095935 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-podinfo\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.096474 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-socket\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.095715 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-credential-socket\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.096734 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8ad439-95c6-4274-8577-4719de10fb3f-istiod-ca-cert\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.096761 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-certs\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.097703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.097167 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-data\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.098777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.098753 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-envoy\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.099523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.099503 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-podinfo\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.161697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.160991 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-token\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.161879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.161712 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7nf5\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-kube-api-access-m7nf5\") pod \"e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.416062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.416037 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:48.861878 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:48.861841 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8ad439_95c6_4274_8577_4719de10fb3f.slice/crio-5d2a6fac470a9ce3baabcc0fc064e478dc2f182c724c08a5a1cea7298c9f39e6 WatchSource:0}: Error finding container 5d2a6fac470a9ce3baabcc0fc064e478dc2f182c724c08a5a1cea7298c9f39e6: Status 404 returned error can't find the container with id 5d2a6fac470a9ce3baabcc0fc064e478dc2f182c724c08a5a1cea7298c9f39e6 Oct 31 00:30:48.863127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.863105 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76"] Oct 31 00:30:48.864820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.864787 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:48.864915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.864861 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:48.864915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:48.864894 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:30:49.045683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:49.045634 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" event={"ID":"2a8ad439-95c6-4274-8577-4719de10fb3f","Type":"ContainerStarted","Data":"66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8"} Oct 31 00:30:49.046153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:49.045720 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" event={"ID":"2a8ad439-95c6-4274-8577-4719de10fb3f","Type":"ContainerStarted","Data":"5d2a6fac470a9ce3baabcc0fc064e478dc2f182c724c08a5a1cea7298c9f39e6"} Oct 31 00:30:49.168259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:49.168200 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" podStartSLOduration=2.168185989 podStartE2EDuration="2.168185989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:49.16583012 +0000 UTC m=+1681.641188542" watchObservedRunningTime="2026-10-31 00:30:49.168185989 +0000 UTC m=+1681.643544411" Oct 31 00:30:49.416917 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:49.416890 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:49.423434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:49.423373 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:49.737046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:49.736967 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be76a6b9-2442-4bdf-bf86-ed3ba919a6e5" path="/var/lib/kubelet/pods/be76a6b9-2442-4bdf-bf86-ed3ba919a6e5/volumes" Oct 31 00:30:50.057371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:50.057300 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:50.058237 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:50.058212 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:30:51.372459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.372422 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs"] Oct 31 00:30:51.378632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.378605 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:30:51.415780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.415751 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs"] Oct 31 00:30:51.439518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.439483 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2a59a7\"" Oct 31 00:30:51.458743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.458714 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5752d604-f290-4a9d-b73a-4d8852d02864-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs\" (UID: \"5752d604-f290-4a9d-b73a-4d8852d02864\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:30:51.458994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.458977 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8jj9\" (UniqueName: \"kubernetes.io/projected/5752d604-f290-4a9d-b73a-4d8852d02864-kube-api-access-q8jj9\") pod \"payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs\" (UID: \"5752d604-f290-4a9d-b73a-4d8852d02864\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:30:51.486401 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.486381 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2a59a7-dockercfg-jr8qk\"" Oct 31 00:30:51.560067 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.560035 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5752d604-f290-4a9d-b73a-4d8852d02864-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs\" (UID: \"5752d604-f290-4a9d-b73a-4d8852d02864\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:30:51.560271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.560257 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8jj9\" (UniqueName: \"kubernetes.io/projected/5752d604-f290-4a9d-b73a-4d8852d02864-kube-api-access-q8jj9\") pod \"payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs\" (UID: \"5752d604-f290-4a9d-b73a-4d8852d02864\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:30:51.560898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.560880 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5752d604-f290-4a9d-b73a-4d8852d02864-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs\" (UID: \"5752d604-f290-4a9d-b73a-4d8852d02864\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:30:51.621798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.621768 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8jj9\" (UniqueName: \"kubernetes.io/projected/5752d604-f290-4a9d-b73a-4d8852d02864-kube-api-access-q8jj9\") pod \"payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs\" (UID: \"5752d604-f290-4a9d-b73a-4d8852d02864\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:30:51.701350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:51.701267 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:30:52.051305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:52.051229 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw"] Oct 31 00:30:52.056124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:52.056102 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98daaf6f-zzpqw"] Oct 31 00:30:52.089442 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:52.089413 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs"] Oct 31 00:30:52.126971 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:52.126941 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5752d604_f290_4a9d_b73a_4d8852d02864.slice/crio-8b253b8e1e89d4b276a0429d4d615dad55cee771b0a987922458c2393b91ddb1 WatchSource:0}: Error finding container 8b253b8e1e89d4b276a0429d4d615dad55cee771b0a987922458c2393b91ddb1: Status 404 returned error can't find the container with id 8b253b8e1e89d4b276a0429d4d615dad55cee771b0a987922458c2393b91ddb1 Oct 31 00:30:53.091839 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:53.091797 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" event={"ID":"5752d604-f290-4a9d-b73a-4d8852d02864","Type":"ContainerStarted","Data":"566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998"} Oct 31 00:30:53.092336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:53.091890 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" event={"ID":"5752d604-f290-4a9d-b73a-4d8852d02864","Type":"ContainerStarted","Data":"8b253b8e1e89d4b276a0429d4d615dad55cee771b0a987922458c2393b91ddb1"} Oct 31 00:30:53.092336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:53.091918 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:30:53.211846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:53.211796 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" podStartSLOduration=2.211781958 podStartE2EDuration="2.211781958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:53.208929024 +0000 UTC m=+1685.684287447" watchObservedRunningTime="2026-10-31 00:30:53.211781958 +0000 UTC m=+1685.687140380" Oct 31 00:30:53.735883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:53.735852 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37c7b3b-c3b5-42a0-97b7-55827c224955" path="/var/lib/kubelet/pods/b37c7b3b-c3b5-42a0-97b7-55827c224955/volumes" Oct 31 00:30:58.056619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.055391 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:30:58.057129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.056829 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:30:58.326440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.326362 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7"] Oct 31 00:30:58.336220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.336191 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:30:58.362061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.362028 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sstj\" (UniqueName: \"kubernetes.io/projected/01767835-222a-4076-b9f3-765ad68b4af8-kube-api-access-5sstj\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7\" (UID: \"01767835-222a-4076-b9f3-765ad68b4af8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:30:58.362302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.362286 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01767835-222a-4076-b9f3-765ad68b4af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7\" (UID: \"01767835-222a-4076-b9f3-765ad68b4af8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:30:58.389656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.389622 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7"] Oct 31 00:30:58.390250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.390224 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bba5ed\"" Oct 31 00:30:58.390387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.390366 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bba5ed-dockercfg-qdsm7\"" Oct 31 00:30:58.480619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.476649 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01767835-222a-4076-b9f3-765ad68b4af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7\" (UID: \"01767835-222a-4076-b9f3-765ad68b4af8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:30:58.480619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.476801 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sstj\" (UniqueName: \"kubernetes.io/projected/01767835-222a-4076-b9f3-765ad68b4af8-kube-api-access-5sstj\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7\" (UID: \"01767835-222a-4076-b9f3-765ad68b4af8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:30:58.563078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.559680 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01767835-222a-4076-b9f3-765ad68b4af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7\" (UID: \"01767835-222a-4076-b9f3-765ad68b4af8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:30:58.633827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.633784 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sstj\" (UniqueName: \"kubernetes.io/projected/01767835-222a-4076-b9f3-765ad68b4af8-kube-api-access-5sstj\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7\" (UID: \"01767835-222a-4076-b9f3-765ad68b4af8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:30:58.652478 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:58.652446 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:30:59.047965 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:30:59.047933 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01767835_222a_4076_b9f3_765ad68b4af8.slice/crio-c71915c36c49875570840a69a9ae736c69040b955f60c97b3c57ca35dc64398e WatchSource:0}: Error finding container c71915c36c49875570840a69a9ae736c69040b955f60c97b3c57ca35dc64398e: Status 404 returned error can't find the container with id c71915c36c49875570840a69a9ae736c69040b955f60c97b3c57ca35dc64398e Oct 31 00:30:59.048941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.048916 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7"] Oct 31 00:30:59.161362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.161329 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" event={"ID":"01767835-222a-4076-b9f3-765ad68b4af8","Type":"ContainerStarted","Data":"d83db1c273ef6d2f4317378445dc085b1067d572a3fbf6f3ff7815211b794b94"} Oct 31 00:30:59.161716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.161391 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" event={"ID":"01767835-222a-4076-b9f3-765ad68b4af8","Type":"ContainerStarted","Data":"c71915c36c49875570840a69a9ae736c69040b955f60c97b3c57ca35dc64398e"} Oct 31 00:30:59.161716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.161511 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:30:59.279744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.279700 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" podStartSLOduration=1.279685486 podStartE2EDuration="1.279685486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:59.276372243 +0000 UTC m=+1691.751730665" watchObservedRunningTime="2026-10-31 00:30:59.279685486 +0000 UTC m=+1691.755043908" Oct 31 00:30:59.494116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.494086 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh"] Oct 31 00:30:59.499566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.499545 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.552886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.552857 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:30:59.553045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.552927 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:30:59.558376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.558346 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh"] Oct 31 00:30:59.598298 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.598269 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed9b59b3-vg2mh\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.598425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.598302 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzj7h\" (UniqueName: \"kubernetes.io/projected/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-kube-api-access-pzj7h\") pod \"maas-api-revoke-sub-ed9b59b3-vg2mh\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.598468 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.598454 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed9b59b3-vg2mh\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.700203 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.700174 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed9b59b3-vg2mh\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.700350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.700336 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed9b59b3-vg2mh\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.700400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.700358 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzj7h\" (UniqueName: \"kubernetes.io/projected/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-kube-api-access-pzj7h\") pod \"maas-api-revoke-sub-ed9b59b3-vg2mh\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.701068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.701045 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed9b59b3-vg2mh\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.702851 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.702827 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed9b59b3-vg2mh\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.761096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.761031 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzj7h\" (UniqueName: \"kubernetes.io/projected/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-kube-api-access-pzj7h\") pod \"maas-api-revoke-sub-ed9b59b3-vg2mh\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:30:59.814883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:30:59.814858 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:31:00.201786 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:31:00.201758 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e92a8d_da40_4da2_b84d_4ebc228fc5d7.slice/crio-e081c94fd27805cd9d351d3cf4fc33a7d612d618866152e77b7a7e27821d60e9 WatchSource:0}: Error finding container e081c94fd27805cd9d351d3cf4fc33a7d612d618866152e77b7a7e27821d60e9: Status 404 returned error can't find the container with id e081c94fd27805cd9d351d3cf4fc33a7d612d618866152e77b7a7e27821d60e9 Oct 31 00:31:00.202400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:00.202340 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh"] Oct 31 00:31:01.180704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:01.180665 4969 generic.go:356] "Generic (PLEG): container finished" podID="b1e92a8d-da40-4da2-b84d-4ebc228fc5d7" containerID="c7147a8b8ccfe2443846aaae120c85c3351e303b899889f37b0eb91f03387a6f" exitCode=0 Oct 31 00:31:01.180901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:01.180738 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" event={"ID":"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7","Type":"ContainerDied","Data":"c7147a8b8ccfe2443846aaae120c85c3351e303b899889f37b0eb91f03387a6f"} Oct 31 00:31:01.180901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:01.180784 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" event={"ID":"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7","Type":"ContainerStarted","Data":"e081c94fd27805cd9d351d3cf4fc33a7d612d618866152e77b7a7e27821d60e9"} Oct 31 00:31:02.051180 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.051142 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8"] Oct 31 00:31:02.100431 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.100396 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9393643-5q2d8"] Oct 31 00:31:02.375080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.375047 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:31:02.450092 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.450059 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-kube-api-access-pzj7h\" (UniqueName: \"kubernetes.io/projected/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-kube-api-access-pzj7h\") pod \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " Oct 31 00:31:02.450337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.450320 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-openshift-ca\") pod \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " Oct 31 00:31:02.450411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.450371 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-xks-ca\") pod \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\" (UID: \"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7\") " Oct 31 00:31:02.450679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.450656 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-openshift-ca" pod "b1e92a8d-da40-4da2-b84d-4ebc228fc5d7" (UID: "b1e92a8d-da40-4da2-b84d-4ebc228fc5d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:02.451173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.451149 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.452496 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.452463 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-kube-api-access-pzj7h" pod "b1e92a8d-da40-4da2-b84d-4ebc228fc5d7" (UID: "b1e92a8d-da40-4da2-b84d-4ebc228fc5d7"). InnerVolumeSpecName "kube-api-access-pzj7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:02.452563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.452533 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-xks-ca" pod "b1e92a8d-da40-4da2-b84d-4ebc228fc5d7" (UID: "b1e92a8d-da40-4da2-b84d-4ebc228fc5d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:02.552419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.552384 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.552419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:02.552413 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzj7h\" (UniqueName: \"kubernetes.io/projected/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7-kube-api-access-pzj7h\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.730493 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:31:02.730462 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:31:03.199830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:03.199799 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" Oct 31 00:31:03.200221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:03.199836 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh" event={"ID":"b1e92a8d-da40-4da2-b84d-4ebc228fc5d7","Type":"ContainerDied","Data":"e081c94fd27805cd9d351d3cf4fc33a7d612d618866152e77b7a7e27821d60e9"} Oct 31 00:31:03.200221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:03.199920 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e081c94fd27805cd9d351d3cf4fc33a7d612d618866152e77b7a7e27821d60e9" Oct 31 00:31:03.736279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:03.736244 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417345c2-7500-4548-9057-f9fbb5081037" path="/var/lib/kubelet/pods/417345c2-7500-4548-9057-f9fbb5081037/volumes" Oct 31 00:31:04.103573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:04.103498 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:31:06.055021 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:06.054978 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns"] Oct 31 00:31:06.098791 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:06.098759 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2846a30b-9rhns"] Oct 31 00:31:07.736692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:07.736660 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13162a1a-fa05-44b7-9ec5-83dd9cd32f4d" path="/var/lib/kubelet/pods/13162a1a-fa05-44b7-9ec5-83dd9cd32f4d/volumes" Oct 31 00:31:10.174197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:10.174168 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:31:16.052472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:16.052428 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4"] Oct 31 00:31:16.108346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:16.108309 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e19166-gqvd4"] Oct 31 00:31:17.737801 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:17.737772 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99c713e-9226-4222-acee-41a5179b29d6" path="/var/lib/kubelet/pods/f99c713e-9226-4222-acee-41a5179b29d6/volumes" Oct 31 00:31:21.051392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:21.051356 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v"] Oct 31 00:31:21.105534 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:21.105497 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a7cfdf-4sb8v"] Oct 31 00:31:21.736733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:21.736695 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d80a0cf-2e96-4d22-b212-10ef2640d5e9" path="/var/lib/kubelet/pods/5d80a0cf-2e96-4d22-b212-10ef2640d5e9/volumes" Oct 31 00:31:24.206828 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:24.206781 4969 patch_prober.go:29] interesting pod/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:31:24.207436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:24.206870 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:31:25.408573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:25.408542 4969 generic.go:356] "Generic (PLEG): container finished" podID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerID="dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939" exitCode=0 Oct 31 00:31:25.409089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:25.408618 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" event={"ID":"dfdec536-f521-4f73-b2ef-8d421a1a688b","Type":"ContainerDied","Data":"dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939"} Oct 31 00:31:25.409136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:25.409122 4969 scope.go:122] "RemoveContainer" containerID="dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939" Oct 31 00:31:25.410828 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:25.410797 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:31:25.410952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:25.410881 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:31:25.410952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:25.410935 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:31:26.420209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:26.420174 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" event={"ID":"dfdec536-f521-4f73-b2ef-8d421a1a688b","Type":"ContainerStarted","Data":"9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b"} Oct 31 00:31:27.255805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:27.255770 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:31:27.255994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:27.255926 4969 patch_prober.go:29] interesting pod/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:31:27.255994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:27.255975 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 00:31:28.255690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:28.255656 4969 patch_prober.go:29] interesting pod/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:31:28.256091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:28.255705 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 00:31:29.255576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:29.255542 4969 patch_prober.go:29] interesting pod/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:31:29.255785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:29.255620 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 00:31:30.255637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:30.255603 4969 patch_prober.go:29] interesting pod/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:31:30.255818 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:30.255653 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 00:31:31.255739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:31.255707 4969 patch_prober.go:29] interesting pod/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 00:31:31.255917 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:31.255760 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 00:31:32.260352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:32.260323 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:31:32.260824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:32.260807 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:31:32.261632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:32.261582 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:31:32.744325 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:31:32.744293 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:31:35.061690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:35.061648 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q"] Oct 31 00:31:35.110464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:35.110431 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2b1248-vfb7q"] Oct 31 00:31:35.737759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:35.737727 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4be4481-060e-4509-b829-5b765f345474" path="/var/lib/kubelet/pods/b4be4481-060e-4509-b829-5b765f345474/volumes" Oct 31 00:31:37.053169 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:37.053122 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4"] Oct 31 00:31:37.058492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:37.058462 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e22c8b6-zmlh4"] Oct 31 00:31:37.737509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:37.737475 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2542c347-1029-47fe-9acb-ded6f13945b2" path="/var/lib/kubelet/pods/2542c347-1029-47fe-9acb-ded6f13945b2/volumes" Oct 31 00:31:38.418083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.418043 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx"] Oct 31 00:31:38.418995 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.418973 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b1e92a8d-da40-4da2-b84d-4ebc228fc5d7" containerName="revoke-keys" Oct 31 00:31:38.418995 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.418996 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e92a8d-da40-4da2-b84d-4ebc228fc5d7" containerName="revoke-keys" Oct 31 00:31:38.419236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.419221 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e92a8d-da40-4da2-b84d-4ebc228fc5d7" containerName="revoke-keys" Oct 31 00:31:38.422748 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.422726 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.481151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.481124 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx"] Oct 31 00:31:38.513271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.513242 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqsv\" (UniqueName: \"kubernetes.io/projected/e451a9be-f1e8-4fcb-b911-15beb654c27b-kube-api-access-clqsv\") pod \"maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.513462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.513444 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.513551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.513537 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.523389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.523367 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:38.523523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.523493 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:31:38.614820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.614789 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqsv\" (UniqueName: \"kubernetes.io/projected/e451a9be-f1e8-4fcb-b911-15beb654c27b-kube-api-access-clqsv\") pod \"maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.615046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.615022 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.615130 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.615113 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.615820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.615792 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.618106 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.618083 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.676130 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.676074 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqsv\" (UniqueName: \"kubernetes.io/projected/e451a9be-f1e8-4fcb-b911-15beb654c27b-kube-api-access-clqsv\") pod \"maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:38.739775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:38.739748 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:39.123927 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:31:39.123899 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode451a9be_f1e8_4fcb_b911_15beb654c27b.slice/crio-49c9160d727df6029bd5ff17a214869ee16a48a3a5a90858aec5c41b9d762031 WatchSource:0}: Error finding container 49c9160d727df6029bd5ff17a214869ee16a48a3a5a90858aec5c41b9d762031: Status 404 returned error can't find the container with id 49c9160d727df6029bd5ff17a214869ee16a48a3a5a90858aec5c41b9d762031 Oct 31 00:31:39.124623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:39.124573 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx"] Oct 31 00:31:39.558037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:39.558000 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" event={"ID":"e451a9be-f1e8-4fcb-b911-15beb654c27b","Type":"ContainerStarted","Data":"49c9160d727df6029bd5ff17a214869ee16a48a3a5a90858aec5c41b9d762031"} Oct 31 00:31:40.568719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:40.568630 4969 generic.go:356] "Generic (PLEG): container finished" podID="e451a9be-f1e8-4fcb-b911-15beb654c27b" containerID="e6e5fd20c60ac072d1acee056a3aa4586d09ab398573e7b16c98b52af0f45596" exitCode=0 Oct 31 00:31:40.569095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:40.568720 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" event={"ID":"e451a9be-f1e8-4fcb-b911-15beb654c27b","Type":"ContainerDied","Data":"e6e5fd20c60ac072d1acee056a3aa4586d09ab398573e7b16c98b52af0f45596"} Oct 31 00:31:41.765221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.765186 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:41.863437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.863351 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-openshift-ca\") pod \"e451a9be-f1e8-4fcb-b911-15beb654c27b\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " Oct 31 00:31:41.863437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.863424 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e451a9be-f1e8-4fcb-b911-15beb654c27b-kube-api-access-clqsv\" (UniqueName: \"kubernetes.io/projected/e451a9be-f1e8-4fcb-b911-15beb654c27b-kube-api-access-clqsv\") pod \"e451a9be-f1e8-4fcb-b911-15beb654c27b\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " Oct 31 00:31:41.863728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.863518 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-xks-ca\") pod \"e451a9be-f1e8-4fcb-b911-15beb654c27b\" (UID: \"e451a9be-f1e8-4fcb-b911-15beb654c27b\") " Oct 31 00:31:41.863799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.863755 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-openshift-ca" pod "e451a9be-f1e8-4fcb-b911-15beb654c27b" (UID: "e451a9be-f1e8-4fcb-b911-15beb654c27b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:41.864168 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.864145 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:31:41.865787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.865737 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e451a9be-f1e8-4fcb-b911-15beb654c27b-kube-api-access-clqsv" pod "e451a9be-f1e8-4fcb-b911-15beb654c27b" (UID: "e451a9be-f1e8-4fcb-b911-15beb654c27b"). InnerVolumeSpecName "kube-api-access-clqsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:41.865867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.865791 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-xks-ca" pod "e451a9be-f1e8-4fcb-b911-15beb654c27b" (UID: "e451a9be-f1e8-4fcb-b911-15beb654c27b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:41.966087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.966046 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clqsv\" (UniqueName: \"kubernetes.io/projected/e451a9be-f1e8-4fcb-b911-15beb654c27b-kube-api-access-clqsv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:31:41.966087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:41.966077 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e451a9be-f1e8-4fcb-b911-15beb654c27b-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:31:42.586871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:42.586840 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" Oct 31 00:31:42.587079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:42.586877 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx" event={"ID":"e451a9be-f1e8-4fcb-b911-15beb654c27b","Type":"ContainerDied","Data":"49c9160d727df6029bd5ff17a214869ee16a48a3a5a90858aec5c41b9d762031"} Oct 31 00:31:42.587079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:42.586930 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49c9160d727df6029bd5ff17a214869ee16a48a3a5a90858aec5c41b9d762031" Oct 31 00:31:47.051015 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:47.050982 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs"] Oct 31 00:31:47.097069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:47.097017 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94d7f41a-6bfcs"] Oct 31 00:31:47.739879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:47.739850 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc571969-762d-4fad-8bd5-0cf8bbdf0755" path="/var/lib/kubelet/pods/fc571969-762d-4fad-8bd5-0cf8bbdf0755/volumes" Oct 31 00:31:48.434625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:48.434563 4969 scope.go:122] "RemoveContainer" containerID="8277dec9d1c7906cf0eec334a5817e154105d8e5833a5eeee645e67482c55c97" Oct 31 00:31:48.445926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:48.445904 4969 scope.go:122] "RemoveContainer" containerID="23e1c219c1196936966a4b8577afde9e4d653342b2da8fafc1051b999db407bc" Oct 31 00:31:48.456354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:48.456336 4969 scope.go:122] "RemoveContainer" containerID="c8b69aa06a5d79dc9c1a04120e0c98206972cd239b8dfce660639f0c65d30f7d" Oct 31 00:31:48.466550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:48.466533 4969 scope.go:122] "RemoveContainer" containerID="e3dd35228f5e56597bf0aa7fe03c570e3a3d3ce8cf7fad4d59227143b8f9f4fd" Oct 31 00:31:48.476466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:48.476449 4969 scope.go:122] "RemoveContainer" containerID="62e2cde4ca5870e8ba42327c6449aa8769bacae0af80ecc42280d140118e2e75" Oct 31 00:31:56.049795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:56.049761 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz"] Oct 31 00:31:56.105585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:56.105552 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0010c9c1-rlnsz"] Oct 31 00:31:57.738307 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:31:57.738267 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc036826-6054-4c9d-92f7-40500e541871" path="/var/lib/kubelet/pods/fc036826-6054-4c9d-92f7-40500e541871/volumes" Oct 31 00:32:01.793242 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:01.793209 4969 generic.go:356] "Generic (PLEG): container finished" podID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerID="501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002" exitCode=0 Oct 31 00:32:01.793686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:01.793285 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" event={"ID":"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611","Type":"ContainerDied","Data":"501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002"} Oct 31 00:32:01.793850 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:01.793832 4969 scope.go:122] "RemoveContainer" containerID="501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002" Oct 31 00:32:01.795492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:01.795465 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:32:01.795542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:01.795533 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:32:01.795581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:01.795568 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:32:02.732413 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:32:02.732379 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:32:02.811940 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:02.811903 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" event={"ID":"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611","Type":"ContainerStarted","Data":"2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec"} Oct 31 00:32:03.501609 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:03.501566 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:32:03.502283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:03.501933 4969 patch_prober.go:29] interesting pod/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:32:03.502482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:03.502460 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 00:32:04.501986 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:04.501945 4969 patch_prober.go:29] interesting pod/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:32:04.502508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:04.502009 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 00:32:05.345646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:05.345547 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs"] Oct 31 00:32:05.406087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:05.406044 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7c92a85-2lrxs"] Oct 31 00:32:05.501976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:05.501938 4969 patch_prober.go:29] interesting pod/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:32:05.502153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:05.502007 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 00:32:05.736919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:05.736885 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1" path="/var/lib/kubelet/pods/bfdf5a3d-bf3e-49fc-a1c0-b0be0ee8e0c1/volumes" Oct 31 00:32:06.501741 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:06.501702 4969 patch_prober.go:29] interesting pod/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:32:06.501973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:06.501782 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 00:32:07.501619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:07.501004 4969 patch_prober.go:29] interesting pod/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 31 00:32:07.501619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:07.501064 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 31 00:32:08.504835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:08.504800 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:32:08.505256 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:08.505217 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:32:08.506118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:08.506095 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:32:17.050908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:17.050875 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd"] Oct 31 00:32:17.100151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:17.100117 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d786b006-c4jtd"] Oct 31 00:32:17.737719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:17.737690 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339bf8b5-e3a2-4db5-b15a-5df2b2198bef" path="/var/lib/kubelet/pods/339bf8b5-e3a2-4db5-b15a-5df2b2198bef/volumes" Oct 31 00:32:20.330693 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.330659 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b"] Oct 31 00:32:20.331610 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.331574 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e451a9be-f1e8-4fcb-b911-15beb654c27b" containerName="revoke-keys" Oct 31 00:32:20.331742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.331614 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e451a9be-f1e8-4fcb-b911-15beb654c27b" containerName="revoke-keys" Oct 31 00:32:20.331824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.331809 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e451a9be-f1e8-4fcb-b911-15beb654c27b" containerName="revoke-keys" Oct 31 00:32:20.334633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.334586 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:20.392227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.392171 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b"] Oct 31 00:32:20.442464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.442432 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f7c92a85-dockercfg-v8zb4\"" Oct 31 00:32:20.444225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.443063 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f7c92a85\"" Oct 31 00:32:20.448034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.448004 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwxgq\" (UniqueName: \"kubernetes.io/projected/d6934960-0fb3-42d3-9506-586edae56ad0-kube-api-access-mwxgq\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b\" (UID: \"d6934960-0fb3-42d3-9506-586edae56ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:20.448155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.448091 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6934960-0fb3-42d3-9506-586edae56ad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b\" (UID: \"d6934960-0fb3-42d3-9506-586edae56ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:20.549736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.549686 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwxgq\" (UniqueName: \"kubernetes.io/projected/d6934960-0fb3-42d3-9506-586edae56ad0-kube-api-access-mwxgq\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b\" (UID: \"d6934960-0fb3-42d3-9506-586edae56ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:20.549929 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.549810 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6934960-0fb3-42d3-9506-586edae56ad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b\" (UID: \"d6934960-0fb3-42d3-9506-586edae56ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:20.550706 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.550670 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6934960-0fb3-42d3-9506-586edae56ad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b\" (UID: \"d6934960-0fb3-42d3-9506-586edae56ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:20.673408 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.673364 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwxgq\" (UniqueName: \"kubernetes.io/projected/d6934960-0fb3-42d3-9506-586edae56ad0-kube-api-access-mwxgq\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b\" (UID: \"d6934960-0fb3-42d3-9506-586edae56ad0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:20.675692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:20.675661 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:21.159965 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:21.159935 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b"] Oct 31 00:32:21.161470 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:32:21.161438 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6934960_0fb3_42d3_9506_586edae56ad0.slice/crio-e2ea06166539d9e13c3366e2f1c187a41d0b43e8c435fd7d2254fe0f8ee3b3f7 WatchSource:0}: Error finding container e2ea06166539d9e13c3366e2f1c187a41d0b43e8c435fd7d2254fe0f8ee3b3f7: Status 404 returned error can't find the container with id e2ea06166539d9e13c3366e2f1c187a41d0b43e8c435fd7d2254fe0f8ee3b3f7 Oct 31 00:32:22.057175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:22.057131 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" event={"ID":"d6934960-0fb3-42d3-9506-586edae56ad0","Type":"ContainerStarted","Data":"e3f0aff8db4e601eb45f0ea4dd75509d9da7431af3848171dcc28834be9f7b09"} Oct 31 00:32:22.061012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:22.060983 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" event={"ID":"d6934960-0fb3-42d3-9506-586edae56ad0","Type":"ContainerStarted","Data":"e2ea06166539d9e13c3366e2f1c187a41d0b43e8c435fd7d2254fe0f8ee3b3f7"} Oct 31 00:32:22.061209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:22.061195 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:22.182933 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:22.182870 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" podStartSLOduration=2.182853857 podStartE2EDuration="2.182853857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:22.179816559 +0000 UTC m=+1774.655174984" watchObservedRunningTime="2026-10-31 00:32:22.182853857 +0000 UTC m=+1774.658212281" Oct 31 00:32:32.729081 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:32:32.729041 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:32:33.069468 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:33.069383 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:32:42.893086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:42.893052 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz"] Oct 31 00:32:42.896846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:42.896821 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:42.956791 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:42.956762 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz"] Oct 31 00:32:42.998369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:42.998329 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:32:42.998534 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:42.998425 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:32:43.018473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.018439 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5hdx\" (UniqueName: \"kubernetes.io/projected/808dac3b-1afd-496e-923a-80be5c1605be-kube-api-access-f5hdx\") pod \"maas-api-revoke-keys-e2e-api-a-835921-xhdzz\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.018642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.018481 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-835921-xhdzz\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.018724 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.018636 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-835921-xhdzz\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.120002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.119974 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5hdx\" (UniqueName: \"kubernetes.io/projected/808dac3b-1afd-496e-923a-80be5c1605be-kube-api-access-f5hdx\") pod \"maas-api-revoke-keys-e2e-api-a-835921-xhdzz\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.120167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.120012 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-835921-xhdzz\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.120167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.120085 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-835921-xhdzz\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.120895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.120873 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-835921-xhdzz\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.122740 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.122720 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-835921-xhdzz\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.181317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.181248 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5hdx\" (UniqueName: \"kubernetes.io/projected/808dac3b-1afd-496e-923a-80be5c1605be-kube-api-access-f5hdx\") pod \"maas-api-revoke-keys-e2e-api-a-835921-xhdzz\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.210265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.210239 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:43.599047 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:32:43.599017 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808dac3b_1afd_496e_923a_80be5c1605be.slice/crio-2406f67808e1c2ee3ba8dd93ec4edd10db562e8433741b7f5d81cbfe8053526f WatchSource:0}: Error finding container 2406f67808e1c2ee3ba8dd93ec4edd10db562e8433741b7f5d81cbfe8053526f: Status 404 returned error can't find the container with id 2406f67808e1c2ee3ba8dd93ec4edd10db562e8433741b7f5d81cbfe8053526f Oct 31 00:32:43.600498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:43.600472 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz"] Oct 31 00:32:44.328024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:44.327937 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" event={"ID":"808dac3b-1afd-496e-923a-80be5c1605be","Type":"ContainerStarted","Data":"5c7c33b4e77d86edc4213e09e923f5609e8d1140890ba5b06c55f7694048e7e0"} Oct 31 00:32:44.328024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:44.327997 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" event={"ID":"808dac3b-1afd-496e-923a-80be5c1605be","Type":"ContainerStarted","Data":"2406f67808e1c2ee3ba8dd93ec4edd10db562e8433741b7f5d81cbfe8053526f"} Oct 31 00:32:44.444989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:44.444932 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" podStartSLOduration=2.44491861 podStartE2EDuration="2.44491861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:44.443695895 +0000 UTC m=+1796.919054315" watchObservedRunningTime="2026-10-31 00:32:44.44491861 +0000 UTC m=+1796.920277033" Oct 31 00:32:45.339258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:45.339227 4969 generic.go:356] "Generic (PLEG): container finished" podID="808dac3b-1afd-496e-923a-80be5c1605be" containerID="5c7c33b4e77d86edc4213e09e923f5609e8d1140890ba5b06c55f7694048e7e0" exitCode=0 Oct 31 00:32:45.339690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:45.339328 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" event={"ID":"808dac3b-1afd-496e-923a-80be5c1605be","Type":"ContainerDied","Data":"5c7c33b4e77d86edc4213e09e923f5609e8d1140890ba5b06c55f7694048e7e0"} Oct 31 00:32:46.509197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.509169 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:46.573453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.573421 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-openshift-ca\") pod \"808dac3b-1afd-496e-923a-80be5c1605be\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " Oct 31 00:32:46.573663 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.573641 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/808dac3b-1afd-496e-923a-80be5c1605be-kube-api-access-f5hdx\" (UniqueName: \"kubernetes.io/projected/808dac3b-1afd-496e-923a-80be5c1605be-kube-api-access-f5hdx\") pod \"808dac3b-1afd-496e-923a-80be5c1605be\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " Oct 31 00:32:46.573726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.573683 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-xks-ca\") pod \"808dac3b-1afd-496e-923a-80be5c1605be\" (UID: \"808dac3b-1afd-496e-923a-80be5c1605be\") " Oct 31 00:32:46.573767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.573744 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-openshift-ca" pod "808dac3b-1afd-496e-923a-80be5c1605be" (UID: "808dac3b-1afd-496e-923a-80be5c1605be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:46.574137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.574124 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:46.576026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.576003 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-xks-ca" pod "808dac3b-1afd-496e-923a-80be5c1605be" (UID: "808dac3b-1afd-496e-923a-80be5c1605be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:46.576100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.576056 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808dac3b-1afd-496e-923a-80be5c1605be-kube-api-access-f5hdx" pod "808dac3b-1afd-496e-923a-80be5c1605be" (UID: "808dac3b-1afd-496e-923a-80be5c1605be"). InnerVolumeSpecName "kube-api-access-f5hdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:46.675968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.675929 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/808dac3b-1afd-496e-923a-80be5c1605be-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:46.675968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:46.675970 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5hdx\" (UniqueName: \"kubernetes.io/projected/808dac3b-1afd-496e-923a-80be5c1605be-kube-api-access-f5hdx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:47.359359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:47.359319 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" Oct 31 00:32:47.359359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:47.359354 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz" event={"ID":"808dac3b-1afd-496e-923a-80be5c1605be","Type":"ContainerDied","Data":"2406f67808e1c2ee3ba8dd93ec4edd10db562e8433741b7f5d81cbfe8053526f"} Oct 31 00:32:47.359580 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:47.359400 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2406f67808e1c2ee3ba8dd93ec4edd10db562e8433741b7f5d81cbfe8053526f" Oct 31 00:32:48.024105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.024076 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:32:48.036695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.036668 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:32:48.045814 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.045790 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:32:48.052607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.052566 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:32:48.548632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.548558 4969 scope.go:122] "RemoveContainer" containerID="040ada1cd33cbeb14320ae4b92e6fe65471fb0b092b1b996b6e6881f7a6a1054" Oct 31 00:32:48.560044 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.560019 4969 scope.go:122] "RemoveContainer" containerID="c707725cc214ae65cb47aae5a4dbe3eb93ddeb0f0d467b8f5a5dba6bc22fa51e" Oct 31 00:32:48.570416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.570395 4969 scope.go:122] "RemoveContainer" containerID="823fd5237d79f6b52a1c0a7965dfbd8cb21ddb1e0596c79fd30bc2f84e710242" Oct 31 00:32:48.580857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.580829 4969 scope.go:122] "RemoveContainer" containerID="110d6d702c2354ceffc2086bce3758b716b58e0b2445f6905fa639ff48718650" Oct 31 00:32:48.590854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.590833 4969 scope.go:122] "RemoveContainer" containerID="8adf5e5ca13cc33e92ba3a6baec73f7414c9f1ae76deea277373f36fb1d6a96d" Oct 31 00:32:48.601169 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.601146 4969 scope.go:122] "RemoveContainer" containerID="6f0b5728a0137083d1f0bcb2b98890d02e547757a3714761b835109cfa12fa3d" Oct 31 00:32:48.611142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.611118 4969 scope.go:122] "RemoveContainer" containerID="36f9b8902f9497cee9c2f9a656c48355418f49dd41752b39b19c185f1bec2bb6" Oct 31 00:32:48.621158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:48.621137 4969 scope.go:122] "RemoveContainer" containerID="52e57bd9112aae4d3df264c96c0cb9717952c645f2326065d8b54fd17f477279" Oct 31 00:32:50.488024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:50.487991 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj"] Oct 31 00:32:50.488427 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:50.488349 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" containerID="cri-o://6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa" gracePeriod=30 Oct 31 00:32:50.526852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:50.526815 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h"] Oct 31 00:32:50.527109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:50.527081 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" containerID="cri-o://3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735" gracePeriod=30 Oct 31 00:32:50.531938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:50.531905 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9"] Oct 31 00:32:50.532150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:50.532120 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" containerID="cri-o://f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1" gracePeriod=30 Oct 31 00:32:50.537023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:50.536998 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw"] Oct 31 00:32:50.537236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:50.537197 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" containerID="cri-o://18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c" gracePeriod=30 Oct 31 00:32:55.564382 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.564346 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6"] Oct 31 00:32:55.617225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.617186 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:32:55.617384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.617277 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:32:55.722924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.720322 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt"] Oct 31 00:32:55.722924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.720626 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" containerID="cri-o://12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649" gracePeriod=30 Oct 31 00:32:55.754913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.750548 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2"] Oct 31 00:32:55.754913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.750825 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" containerID="cri-o://b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc" gracePeriod=30 Oct 31 00:32:55.764767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.764689 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6"] Oct 31 00:32:55.766016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.765989 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c"] Oct 31 00:32:55.766392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.766340 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" containerID="cri-o://c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471" gracePeriod=30 Oct 31 00:32:55.781617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:55.772743 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" containerID="cri-o://4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8" gracePeriod=30 Oct 31 00:32:56.039611 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.038814 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146301 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-data\") pod \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146446 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-certs\") pod \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146512 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-credential-socket\") pod \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146560 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-data" pod "f32982e9-15ef-45ac-ab4c-55f742b5bc4c" (UID: "f32982e9-15ef-45ac-ab4c-55f742b5bc4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146571 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-token\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-token\") pod \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146659 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-kube-api-access-gf8r6\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-kube-api-access-gf8r6\") pod \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146761 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-certs" pod "f32982e9-15ef-45ac-ab4c-55f742b5bc4c" (UID: "f32982e9-15ef-45ac-ab4c-55f742b5bc4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146767 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-podinfo\") pod \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146866 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-envoy\") pod \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146939 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istiod-ca-cert\") pod \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.146992 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-socket\") pod \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\" (UID: \"f32982e9-15ef-45ac-ab4c-55f742b5bc4c\") " Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.147319 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-credential-socket" pod "f32982e9-15ef-45ac-ab4c-55f742b5bc4c" (UID: "f32982e9-15ef-45ac-ab4c-55f742b5bc4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.147494 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-socket" pod "f32982e9-15ef-45ac-ab4c-55f742b5bc4c" (UID: "f32982e9-15ef-45ac-ab4c-55f742b5bc4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.147808 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.147831 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.147847 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.147862 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.148157 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istiod-ca-cert" pod "f32982e9-15ef-45ac-ab4c-55f742b5bc4c" (UID: "f32982e9-15ef-45ac-ab4c-55f742b5bc4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.152008 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-token" pod "f32982e9-15ef-45ac-ab4c-55f742b5bc4c" (UID: "f32982e9-15ef-45ac-ab4c-55f742b5bc4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.153606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.152053 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-podinfo" pod "f32982e9-15ef-45ac-ab4c-55f742b5bc4c" (UID: "f32982e9-15ef-45ac-ab4c-55f742b5bc4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:32:56.158736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.156049 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-envoy" pod "f32982e9-15ef-45ac-ab4c-55f742b5bc4c" (UID: "f32982e9-15ef-45ac-ab4c-55f742b5bc4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.175747 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.175714 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-kube-api-access-gf8r6" pod "f32982e9-15ef-45ac-ab4c-55f742b5bc4c" (UID: "f32982e9-15ef-45ac-ab4c-55f742b5bc4c"). InnerVolumeSpecName "kube-api-access-gf8r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.221483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.221460 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:32:56.225562 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.225537 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:32:56.229820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.229801 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:32:56.249077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.249056 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.249077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.249079 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf8r6\" (UniqueName: \"kubernetes.io/projected/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-kube-api-access-gf8r6\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.249219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.249090 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.249219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.249099 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.249219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.249108 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32982e9-15ef-45ac-ab4c-55f742b5bc4c-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.350244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350160 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-envoy\") pod \"632906a4-9437-49ae-b08c-a67e11cb6cb7\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " Oct 31 00:32:56.350244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350217 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-data\") pod \"632906a4-9437-49ae-b08c-a67e11cb6cb7\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " Oct 31 00:32:56.350445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350281 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-certs\") pod \"632906a4-9437-49ae-b08c-a67e11cb6cb7\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " Oct 31 00:32:56.350445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350305 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-envoy\") pod \"64cf045f-a7b5-4051-92a4-67956ad461a4\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " Oct 31 00:32:56.350445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350335 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-kube-api-access-9pc4z\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-kube-api-access-9pc4z\") pod \"632906a4-9437-49ae-b08c-a67e11cb6cb7\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " Oct 31 00:32:56.350445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350369 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64cf045f-a7b5-4051-92a4-67956ad461a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf045f-a7b5-4051-92a4-67956ad461a4-istiod-ca-cert\") pod \"64cf045f-a7b5-4051-92a4-67956ad461a4\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " Oct 31 00:32:56.350445 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350401 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-credential-socket\") pod \"632906a4-9437-49ae-b08c-a67e11cb6cb7\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350446 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-certs\") pod \"895db242-19e2-4882-a8f5-2375cc5a34c5\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350482 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-token\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-token\") pod \"64cf045f-a7b5-4051-92a4-67956ad461a4\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350504 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-data" pod "632906a4-9437-49ae-b08c-a67e11cb6cb7" (UID: "632906a4-9437-49ae-b08c-a67e11cb6cb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350520 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-envoy\") pod \"895db242-19e2-4882-a8f5-2375cc5a34c5\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350551 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-credential-socket\") pod \"895db242-19e2-4882-a8f5-2375cc5a34c5\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350585 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-token\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-token\") pod \"632906a4-9437-49ae-b08c-a67e11cb6cb7\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350691 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-podinfo\") pod \"64cf045f-a7b5-4051-92a4-67956ad461a4\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350743 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-kube-api-access-z269l\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-kube-api-access-z269l\") pod \"895db242-19e2-4882-a8f5-2375cc5a34c5\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350783 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-token\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-token\") pod \"895db242-19e2-4882-a8f5-2375cc5a34c5\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " Oct 31 00:32:56.350809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350808 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-certs" pod "895db242-19e2-4882-a8f5-2375cc5a34c5" (UID: "895db242-19e2-4882-a8f5-2375cc5a34c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350824 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-socket\") pod \"64cf045f-a7b5-4051-92a4-67956ad461a4\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350832 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cf045f-a7b5-4051-92a4-67956ad461a4-istiod-ca-cert" pod "64cf045f-a7b5-4051-92a4-67956ad461a4" (UID: "64cf045f-a7b5-4051-92a4-67956ad461a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.350888 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895db242-19e2-4882-a8f5-2375cc5a34c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db242-19e2-4882-a8f5-2375cc5a34c5-istiod-ca-cert\") pod \"895db242-19e2-4882-a8f5-2375cc5a34c5\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351011 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-socket" pod "64cf045f-a7b5-4051-92a4-67956ad461a4" (UID: "64cf045f-a7b5-4051-92a4-67956ad461a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351028 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-certs\") pod \"64cf045f-a7b5-4051-92a4-67956ad461a4\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351039 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-credential-socket" pod "632906a4-9437-49ae-b08c-a67e11cb6cb7" (UID: "632906a4-9437-49ae-b08c-a67e11cb6cb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351059 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-data\") pod \"64cf045f-a7b5-4051-92a4-67956ad461a4\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351100 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-kube-api-access-vvh99\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-kube-api-access-vvh99\") pod \"64cf045f-a7b5-4051-92a4-67956ad461a4\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351161 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-credential-socket\") pod \"64cf045f-a7b5-4051-92a4-67956ad461a4\" (UID: \"64cf045f-a7b5-4051-92a4-67956ad461a4\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351238 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-data\") pod \"895db242-19e2-4882-a8f5-2375cc5a34c5\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351278 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-socket\") pod \"895db242-19e2-4882-a8f5-2375cc5a34c5\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351397 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-podinfo\") pod \"895db242-19e2-4882-a8f5-2375cc5a34c5\" (UID: \"895db242-19e2-4882-a8f5-2375cc5a34c5\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351430 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/632906a4-9437-49ae-b08c-a67e11cb6cb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632906a4-9437-49ae-b08c-a67e11cb6cb7-istiod-ca-cert\") pod \"632906a4-9437-49ae-b08c-a67e11cb6cb7\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " Oct 31 00:32:56.351498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351490 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-data" pod "64cf045f-a7b5-4051-92a4-67956ad461a4" (UID: "64cf045f-a7b5-4051-92a4-67956ad461a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.352520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351501 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-podinfo\") pod \"632906a4-9437-49ae-b08c-a67e11cb6cb7\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " Oct 31 00:32:56.352520 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351573 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-socket\") pod \"632906a4-9437-49ae-b08c-a67e11cb6cb7\" (UID: \"632906a4-9437-49ae-b08c-a67e11cb6cb7\") " Oct 31 00:32:56.352661 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.352571 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.352661 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.352655 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.352765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.352672 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64cf045f-a7b5-4051-92a4-67956ad461a4-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.352765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.352689 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.352765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.352703 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.352765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.352732 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.355181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.351514 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-credential-socket" pod "895db242-19e2-4882-a8f5-2375cc5a34c5" (UID: "895db242-19e2-4882-a8f5-2375cc5a34c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.353260 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-certs" pod "632906a4-9437-49ae-b08c-a67e11cb6cb7" (UID: "632906a4-9437-49ae-b08c-a67e11cb6cb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.353395 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-socket" pod "895db242-19e2-4882-a8f5-2375cc5a34c5" (UID: "895db242-19e2-4882-a8f5-2375cc5a34c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.353678 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-credential-socket" pod "64cf045f-a7b5-4051-92a4-67956ad461a4" (UID: "64cf045f-a7b5-4051-92a4-67956ad461a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.353959 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-data" pod "895db242-19e2-4882-a8f5-2375cc5a34c5" (UID: "895db242-19e2-4882-a8f5-2375cc5a34c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.354491 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632906a4-9437-49ae-b08c-a67e11cb6cb7-istiod-ca-cert" pod "632906a4-9437-49ae-b08c-a67e11cb6cb7" (UID: "632906a4-9437-49ae-b08c-a67e11cb6cb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.354670 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-podinfo" pod "64cf045f-a7b5-4051-92a4-67956ad461a4" (UID: "64cf045f-a7b5-4051-92a4-67956ad461a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.354782 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-certs" pod "64cf045f-a7b5-4051-92a4-67956ad461a4" (UID: "64cf045f-a7b5-4051-92a4-67956ad461a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.355092 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895db242-19e2-4882-a8f5-2375cc5a34c5-istiod-ca-cert" pod "895db242-19e2-4882-a8f5-2375cc5a34c5" (UID: "895db242-19e2-4882-a8f5-2375cc5a34c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.355147 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-envoy" pod "895db242-19e2-4882-a8f5-2375cc5a34c5" (UID: "895db242-19e2-4882-a8f5-2375cc5a34c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.355279 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-socket" pod "632906a4-9437-49ae-b08c-a67e11cb6cb7" (UID: "632906a4-9437-49ae-b08c-a67e11cb6cb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.355271 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-envoy" pod "632906a4-9437-49ae-b08c-a67e11cb6cb7" (UID: "632906a4-9437-49ae-b08c-a67e11cb6cb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.355545 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-envoy" pod "64cf045f-a7b5-4051-92a4-67956ad461a4" (UID: "64cf045f-a7b5-4051-92a4-67956ad461a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.355922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.355559 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-kube-api-access-9pc4z" pod "632906a4-9437-49ae-b08c-a67e11cb6cb7" (UID: "632906a4-9437-49ae-b08c-a67e11cb6cb7"). InnerVolumeSpecName "kube-api-access-9pc4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.356035 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.356012 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-podinfo" pod "632906a4-9437-49ae-b08c-a67e11cb6cb7" (UID: "632906a4-9437-49ae-b08c-a67e11cb6cb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:32:56.356094 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.356031 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-kube-api-access-z269l" pod "895db242-19e2-4882-a8f5-2375cc5a34c5" (UID: "895db242-19e2-4882-a8f5-2375cc5a34c5"). InnerVolumeSpecName "kube-api-access-z269l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.356094 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.356067 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-kube-api-access-vvh99" pod "64cf045f-a7b5-4051-92a4-67956ad461a4" (UID: "64cf045f-a7b5-4051-92a4-67956ad461a4"). InnerVolumeSpecName "kube-api-access-vvh99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.356341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.356324 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-token" pod "895db242-19e2-4882-a8f5-2375cc5a34c5" (UID: "895db242-19e2-4882-a8f5-2375cc5a34c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.356630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.356611 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-token" pod "632906a4-9437-49ae-b08c-a67e11cb6cb7" (UID: "632906a4-9437-49ae-b08c-a67e11cb6cb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.356752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.356646 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-token" pod "64cf045f-a7b5-4051-92a4-67956ad461a4" (UID: "64cf045f-a7b5-4051-92a4-67956ad461a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.356752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.356714 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-podinfo" pod "895db242-19e2-4882-a8f5-2375cc5a34c5" (UID: "895db242-19e2-4882-a8f5-2375cc5a34c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:32:56.454783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454744 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.454925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454793 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db242-19e2-4882-a8f5-2375cc5a34c5-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.454925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454810 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.454925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454825 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvh99\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-kube-api-access-vvh99\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.454925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454843 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.454925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454857 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.454925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454872 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.454925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454886 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.454925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454900 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632906a4-9437-49ae-b08c-a67e11cb6cb7-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.454925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454914 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454928 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454941 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454958 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632906a4-9437-49ae-b08c-a67e11cb6cb7-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454971 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.454985 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pc4z\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-kube-api-access-9pc4z\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.455000 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.455015 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.455027 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db242-19e2-4882-a8f5-2375cc5a34c5-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.455041 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632906a4-9437-49ae-b08c-a67e11cb6cb7-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.455054 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64cf045f-a7b5-4051-92a4-67956ad461a4-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.455212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.455070 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z269l\" (UniqueName: \"kubernetes.io/projected/895db242-19e2-4882-a8f5-2375cc5a34c5-kube-api-access-z269l\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:32:56.458232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.458200 4969 generic.go:356] "Generic (PLEG): container finished" podID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerID="f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1" exitCode=0 Oct 31 00:32:56.458378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.458259 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" Oct 31 00:32:56.458378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.458320 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" event={"ID":"895db242-19e2-4882-a8f5-2375cc5a34c5","Type":"ContainerDied","Data":"f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1"} Oct 31 00:32:56.458378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.458371 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9" event={"ID":"895db242-19e2-4882-a8f5-2375cc5a34c5","Type":"ContainerDied","Data":"40242c1301ad25f8abdf37018dda4871a90b8a2e6b5373e5fc02fc484b3d6c4f"} Oct 31 00:32:56.458554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.458398 4969 scope.go:122] "RemoveContainer" containerID="f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1" Oct 31 00:32:56.460979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.460950 4969 generic.go:356] "Generic (PLEG): container finished" podID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerID="6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa" exitCode=0 Oct 31 00:32:56.461081 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.461010 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" Oct 31 00:32:56.461081 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.461019 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" event={"ID":"f32982e9-15ef-45ac-ab4c-55f742b5bc4c","Type":"ContainerDied","Data":"6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa"} Oct 31 00:32:56.461200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.461114 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj" event={"ID":"f32982e9-15ef-45ac-ab4c-55f742b5bc4c","Type":"ContainerDied","Data":"c398eadd2d44d3da1977a4d84e045d7495f585957e08db4aa492d2b074d6539f"} Oct 31 00:32:56.462992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.462963 4969 generic.go:356] "Generic (PLEG): container finished" podID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerID="3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735" exitCode=0 Oct 31 00:32:56.463183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.463165 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" event={"ID":"632906a4-9437-49ae-b08c-a67e11cb6cb7","Type":"ContainerDied","Data":"3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735"} Oct 31 00:32:56.463253 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.463209 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" event={"ID":"632906a4-9437-49ae-b08c-a67e11cb6cb7","Type":"ContainerDied","Data":"df54032cc363173c51407a2c9d0fefec471534b1ff7d00df64801facd1bd48bc"} Oct 31 00:32:56.463309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.463286 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h" Oct 31 00:32:56.467204 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.467188 4969 generic.go:356] "Generic (PLEG): container finished" podID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerID="18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c" exitCode=0 Oct 31 00:32:56.467402 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.467247 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" Oct 31 00:32:56.467636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.467258 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" event={"ID":"64cf045f-a7b5-4051-92a4-67956ad461a4","Type":"ContainerDied","Data":"18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c"} Oct 31 00:32:56.467636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.467556 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw" event={"ID":"64cf045f-a7b5-4051-92a4-67956ad461a4","Type":"ContainerDied","Data":"9d3b386aa0e2c5ab598aa31b3305763ae55ce816b72bdfe56e3a655e5312cb9e"} Oct 31 00:32:56.496962 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.489888 4969 scope.go:122] "RemoveContainer" containerID="f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1" Oct 31 00:32:56.512141 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:32:56.512109 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1\": container with ID starting with f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1 not found: ID does not exist" containerID="f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1" Oct 31 00:32:56.512276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.512154 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1"} err="failed to get container status \"f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1\": rpc error: code = NotFound desc = could not find container \"f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1\": container with ID starting with f01259df05563e5b76f74722b2fa5821a9214d02baabd827d25ac073b1a5a1f1 not found: ID does not exist" Oct 31 00:32:56.512276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.512178 4969 scope.go:122] "RemoveContainer" containerID="6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa" Oct 31 00:32:56.517300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.517277 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj"] Oct 31 00:32:56.518190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518168 4969 cpu_manager.go:414] "containerMap: removing container" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" Oct 31 00:32:56.518288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518192 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" Oct 31 00:32:56.518288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518214 4969 cpu_manager.go:414] "containerMap: removing container" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" Oct 31 00:32:56.518288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518222 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" Oct 31 00:32:56.518288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518239 4969 cpu_manager.go:414] "containerMap: removing container" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" Oct 31 00:32:56.518288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518247 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" Oct 31 00:32:56.518288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518263 4969 cpu_manager.go:414] "containerMap: removing container" podUID="808dac3b-1afd-496e-923a-80be5c1605be" containerName="revoke-keys" Oct 31 00:32:56.518288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518270 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="808dac3b-1afd-496e-923a-80be5c1605be" containerName="revoke-keys" Oct 31 00:32:56.518288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518284 4969 cpu_manager.go:414] "containerMap: removing container" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" Oct 31 00:32:56.518288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518291 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" Oct 31 00:32:56.518771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518306 4969 cpu_manager.go:414] "containerMap: removing container" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" Oct 31 00:32:56.518771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518315 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" Oct 31 00:32:56.518771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518489 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" containerName="istio-proxy" Oct 31 00:32:56.518771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518504 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" containerName="istio-proxy" Oct 31 00:32:56.518771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518514 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" Oct 31 00:32:56.518771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518523 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="808dac3b-1afd-496e-923a-80be5c1605be" containerName="revoke-keys" Oct 31 00:32:56.518771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518536 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" containerName="istio-proxy" Oct 31 00:32:56.518771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.518545 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" containerName="istio-proxy" Oct 31 00:32:56.522381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.522325 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.583892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.583835 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-70babe-openshift-default-dockercfg-xzw59\"" Oct 31 00:32:56.601655 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.601089 4969 scope.go:122] "RemoveContainer" containerID="768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b" Oct 31 00:32:56.622954 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.622922 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj"] Oct 31 00:32:56.626386 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.626356 4969 scope.go:122] "RemoveContainer" containerID="6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa" Oct 31 00:32:56.627263 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:32:56.627230 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa\": container with ID starting with 6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa not found: ID does not exist" containerID="6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa" Oct 31 00:32:56.627391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.627276 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa"} err="failed to get container status \"6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa\": rpc error: code = NotFound desc = could not find container \"6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa\": container with ID starting with 6669003088fad45cf1830a1d3d82881cab424b72fa628833fb4204b228b0e5fa not found: ID does not exist" Oct 31 00:32:56.627391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.627299 4969 scope.go:122] "RemoveContainer" containerID="768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b" Oct 31 00:32:56.627649 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:32:56.627625 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b\": container with ID starting with 768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b not found: ID does not exist" containerID="768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b" Oct 31 00:32:56.627742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.627657 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b"} err="failed to get container status \"768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b\": rpc error: code = NotFound desc = could not find container \"768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b\": container with ID starting with 768113837e9fd7e1b1c3d1d3deb66789c17deb3a4cf5d6a27e9467813de0281b not found: ID does not exist" Oct 31 00:32:56.627742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.627678 4969 scope.go:122] "RemoveContainer" containerID="3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735" Oct 31 00:32:56.639395 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.639369 4969 scope.go:122] "RemoveContainer" containerID="3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735" Oct 31 00:32:56.639783 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:32:56.639755 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735\": container with ID starting with 3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735 not found: ID does not exist" containerID="3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735" Oct 31 00:32:56.639884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.639792 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735"} err="failed to get container status \"3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735\": rpc error: code = NotFound desc = could not find container \"3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735\": container with ID starting with 3a4295300b8af200ea1604124364df2a40b0db38ea700e4ee069bdd5b3993735 not found: ID does not exist" Oct 31 00:32:56.639884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.639814 4969 scope.go:122] "RemoveContainer" containerID="18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c" Oct 31 00:32:56.653678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.653653 4969 scope.go:122] "RemoveContainer" containerID="18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c" Oct 31 00:32:56.654054 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:32:56.654030 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c\": container with ID starting with 18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c not found: ID does not exist" containerID="18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c" Oct 31 00:32:56.654164 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.654063 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c"} err="failed to get container status \"18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c\": rpc error: code = NotFound desc = could not find container \"18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c\": container with ID starting with 18df1ba931eb7a7878e3e89f2dd0bdd920a63a9b04b76339d58a36da15a2ac5c not found: ID does not exist" Oct 31 00:32:56.666201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.666160 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f2becb-a214-43c1-8d2c-519a00bfd076-istiod-ca-cert\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.666312 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.666294 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-data\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.666391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.666330 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-token\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.666453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.666435 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-socket\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.666519 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.666457 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-credential-socket\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.666632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.666577 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-certs\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.666697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.666659 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-envoy\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.666759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.666713 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8l8\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-kube-api-access-7c8l8\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.666866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.666804 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-podinfo\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.795757 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-envoy\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.795856 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8l8\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-kube-api-access-7c8l8\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.796015 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-podinfo\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.796071 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f2becb-a214-43c1-8d2c-519a00bfd076-istiod-ca-cert\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.796182 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-data\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.796222 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-token\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.796372 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-socket\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.796401 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-credential-socket\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.796633 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-certs\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.797240 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-certs\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.799512 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-data\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.807520 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-socket\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.807896 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-credential-socket\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.833483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.820504 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f2becb-a214-43c1-8d2c-519a00bfd076-istiod-ca-cert\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.860673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.856884 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-podinfo\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.860673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.857454 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-envoy\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.873835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.865534 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-token\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.894812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.894727 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8l8\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-kube-api-access-7c8l8\") pod \"e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.927967 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.927928 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:56.988683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:56.988647 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj"] Oct 31 00:32:57.001204 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.001167 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lzdgj"] Oct 31 00:32:57.130614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.123908 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw"] Oct 31 00:32:57.166121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.166084 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-ncrxw"] Oct 31 00:32:57.221571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.221531 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h"] Oct 31 00:32:57.230977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.230948 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qsz7h"] Oct 31 00:32:57.346047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.345978 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9"] Oct 31 00:32:57.356023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.355882 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-7c2n9"] Oct 31 00:32:57.392358 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:32:57.392328 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f2becb_a214_43c1_8d2c_519a00bfd076.slice/crio-eff7ef4a12874002c3f92f93d5e2a37d4e0ce2cb4bdd53a43082a5f7144627c9 WatchSource:0}: Error finding container eff7ef4a12874002c3f92f93d5e2a37d4e0ce2cb4bdd53a43082a5f7144627c9: Status 404 returned error can't find the container with id eff7ef4a12874002c3f92f93d5e2a37d4e0ce2cb4bdd53a43082a5f7144627c9 Oct 31 00:32:57.393751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.393723 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj"] Oct 31 00:32:57.395188 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.395156 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:32:57.395288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.395224 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:32:57.395288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.395258 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:32:57.503642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.498387 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" event={"ID":"13f2becb-a214-43c1-8d2c-519a00bfd076","Type":"ContainerStarted","Data":"eff7ef4a12874002c3f92f93d5e2a37d4e0ce2cb4bdd53a43082a5f7144627c9"} Oct 31 00:32:57.738862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.738780 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="632906a4-9437-49ae-b08c-a67e11cb6cb7" path="/var/lib/kubelet/pods/632906a4-9437-49ae-b08c-a67e11cb6cb7/volumes" Oct 31 00:32:57.739308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.739271 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64cf045f-a7b5-4051-92a4-67956ad461a4" path="/var/lib/kubelet/pods/64cf045f-a7b5-4051-92a4-67956ad461a4/volumes" Oct 31 00:32:57.739744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.739723 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895db242-19e2-4882-a8f5-2375cc5a34c5" path="/var/lib/kubelet/pods/895db242-19e2-4882-a8f5-2375cc5a34c5/volumes" Oct 31 00:32:57.740461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:57.740444 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32982e9-15ef-45ac-ab4c-55f742b5bc4c" path="/var/lib/kubelet/pods/f32982e9-15ef-45ac-ab4c-55f742b5bc4c/volumes" Oct 31 00:32:58.348342 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:58.348304 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7"] Oct 31 00:32:58.520731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:58.520379 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" event={"ID":"13f2becb-a214-43c1-8d2c-519a00bfd076","Type":"ContainerStarted","Data":"81464f271916c299164d58c66e6365530f0aeae425794f54e7375d823e51096c"} Oct 31 00:32:58.643083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:58.643032 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" podStartSLOduration=2.6430162299999997 podStartE2EDuration="2.64301623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:58.640382417 +0000 UTC m=+1811.115740843" watchObservedRunningTime="2026-10-31 00:32:58.64301623 +0000 UTC m=+1811.118374654" Oct 31 00:32:58.928888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:58.928796 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:58.935120 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:58.935094 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:59.472683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:59.472650 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:32:59.472895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:59.472704 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:59.530998 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:59.530955 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:32:59.531955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:32:59.531931 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:33:00.565476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:00.565429 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" podUID="98ee07e7-f923-43eb-b1a4-aec71c96013f" containerName="payload-pre-processing" containerID="cri-o://b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831" gracePeriod=25 Oct 31 00:33:01.005898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.005868 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:33:01.099901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.099876 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ee07e7-f923-43eb-b1a4-aec71c96013f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ee07e7-f923-43eb-b1a4-aec71c96013f-plugins-config-volume\") pod \"98ee07e7-f923-43eb-b1a4-aec71c96013f\" (UID: \"98ee07e7-f923-43eb-b1a4-aec71c96013f\") " Oct 31 00:33:01.100030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.099992 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ee07e7-f923-43eb-b1a4-aec71c96013f-kube-api-access-rfn6n\" (UniqueName: \"kubernetes.io/projected/98ee07e7-f923-43eb-b1a4-aec71c96013f-kube-api-access-rfn6n\") pod \"98ee07e7-f923-43eb-b1a4-aec71c96013f\" (UID: \"98ee07e7-f923-43eb-b1a4-aec71c96013f\") " Oct 31 00:33:01.100304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.100279 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ee07e7-f923-43eb-b1a4-aec71c96013f-plugins-config-volume" pod "98ee07e7-f923-43eb-b1a4-aec71c96013f" (UID: "98ee07e7-f923-43eb-b1a4-aec71c96013f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:01.101340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.101316 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ee07e7-f923-43eb-b1a4-aec71c96013f-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.104940 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.104911 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ee07e7-f923-43eb-b1a4-aec71c96013f-kube-api-access-rfn6n" pod "98ee07e7-f923-43eb-b1a4-aec71c96013f" (UID: "98ee07e7-f923-43eb-b1a4-aec71c96013f"). InnerVolumeSpecName "kube-api-access-rfn6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:01.132491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.132455 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:33:01.182042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.181958 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb"] Oct 31 00:33:01.182686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.182666 4969 cpu_manager.go:414] "containerMap: removing container" podUID="98ee07e7-f923-43eb-b1a4-aec71c96013f" containerName="payload-pre-processing" Oct 31 00:33:01.182686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.182685 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ee07e7-f923-43eb-b1a4-aec71c96013f" containerName="payload-pre-processing" Oct 31 00:33:01.182856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.182700 4969 cpu_manager.go:414] "containerMap: removing container" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" Oct 31 00:33:01.182856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.182705 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" Oct 31 00:33:01.182856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.182831 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ee07e7-f923-43eb-b1a4-aec71c96013f" containerName="payload-pre-processing" Oct 31 00:33:01.182856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.182841 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerName="istio-proxy" Oct 31 00:33:01.185411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.185388 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.202163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202144 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79da24b1-7783-4b4e-acb0-df82e0165b87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da24b1-7783-4b4e-acb0-df82e0165b87-istiod-ca-cert\") pod \"79da24b1-7783-4b4e-acb0-df82e0165b87\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " Oct 31 00:33:01.202332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202202 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-envoy\") pod \"79da24b1-7783-4b4e-acb0-df82e0165b87\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " Oct 31 00:33:01.202332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202259 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-socket\") pod \"79da24b1-7783-4b4e-acb0-df82e0165b87\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " Oct 31 00:33:01.202471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202455 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-podinfo\") pod \"79da24b1-7783-4b4e-acb0-df82e0165b87\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " Oct 31 00:33:01.202523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202509 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-token\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-token\") pod \"79da24b1-7783-4b4e-acb0-df82e0165b87\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " Oct 31 00:33:01.202645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202578 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-certs\") pod \"79da24b1-7783-4b4e-acb0-df82e0165b87\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " Oct 31 00:33:01.202704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202687 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-credential-socket\") pod \"79da24b1-7783-4b4e-acb0-df82e0165b87\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " Oct 31 00:33:01.202748 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202726 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-kube-api-access-d6l6j\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-kube-api-access-d6l6j\") pod \"79da24b1-7783-4b4e-acb0-df82e0165b87\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " Oct 31 00:33:01.202801 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202576 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79da24b1-7783-4b4e-acb0-df82e0165b87-istiod-ca-cert" pod "79da24b1-7783-4b4e-acb0-df82e0165b87" (UID: "79da24b1-7783-4b4e-acb0-df82e0165b87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:01.202801 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202775 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-data\") pod \"79da24b1-7783-4b4e-acb0-df82e0165b87\" (UID: \"79da24b1-7783-4b4e-acb0-df82e0165b87\") " Oct 31 00:33:01.203002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.202982 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-data" pod "79da24b1-7783-4b4e-acb0-df82e0165b87" (UID: "79da24b1-7783-4b4e-acb0-df82e0165b87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.203247 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.203194 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-credential-socket" pod "79da24b1-7783-4b4e-acb0-df82e0165b87" (UID: "79da24b1-7783-4b4e-acb0-df82e0165b87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.203479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.203459 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-certs" pod "79da24b1-7783-4b4e-acb0-df82e0165b87" (UID: "79da24b1-7783-4b4e-acb0-df82e0165b87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.203574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.203526 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-socket" pod "79da24b1-7783-4b4e-acb0-df82e0165b87" (UID: "79da24b1-7783-4b4e-acb0-df82e0165b87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.203677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.203659 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.203755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.203685 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.203755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.203700 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.203755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.203715 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da24b1-7783-4b4e-acb0-df82e0165b87-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.203755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.203730 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfn6n\" (UniqueName: \"kubernetes.io/projected/98ee07e7-f923-43eb-b1a4-aec71c96013f-kube-api-access-rfn6n\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.205476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.205455 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-token" pod "79da24b1-7783-4b4e-acb0-df82e0165b87" (UID: "79da24b1-7783-4b4e-acb0-df82e0165b87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:01.205686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.205665 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-kube-api-access-d6l6j" pod "79da24b1-7783-4b4e-acb0-df82e0165b87" (UID: "79da24b1-7783-4b4e-acb0-df82e0165b87"). InnerVolumeSpecName "kube-api-access-d6l6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:01.206392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.206369 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-podinfo" pod "79da24b1-7783-4b4e-acb0-df82e0165b87" (UID: "79da24b1-7783-4b4e-acb0-df82e0165b87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:33:01.207848 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.207821 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-envoy" pod "79da24b1-7783-4b4e-acb0-df82e0165b87" (UID: "79da24b1-7783-4b4e-acb0-df82e0165b87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.211937 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.211914 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:33:01.216981 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.216964 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:33:01.220777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.220752 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:33:01.237818 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.237788 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f7c92a85-serving-cert\"" Oct 31 00:33:01.237945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.237905 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:33:01.237945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.237912 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-442zl\"" Oct 31 00:33:01.255992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.255920 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb"] Oct 31 00:33:01.304491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304458 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b79597fb-1903-4420-a542-1b036dd4e36c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79597fb-1903-4420-a542-1b036dd4e36c-istio-podinfo\") pod \"b79597fb-1903-4420-a542-1b036dd4e36c\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " Oct 31 00:33:01.304691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304539 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-kube-api-access-5d9h8\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-kube-api-access-5d9h8\") pod \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " Oct 31 00:33:01.304691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304576 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-credential-socket\") pod \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " Oct 31 00:33:01.304691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304633 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-data\") pod \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " Oct 31 00:33:01.304691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304653 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-istio-token\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-istio-token\") pod \"b79597fb-1903-4420-a542-1b036dd4e36c\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " Oct 31 00:33:01.304691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304670 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-podinfo\") pod \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " Oct 31 00:33:01.304971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304698 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-envoy\") pod \"b79597fb-1903-4420-a542-1b036dd4e36c\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " Oct 31 00:33:01.304971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304790 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b5e359-4787-45df-b48e-b83fabb5afb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b5e359-4787-45df-b48e-b83fabb5afb9-istiod-ca-cert\") pod \"18b5e359-4787-45df-b48e-b83fabb5afb9\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " Oct 31 00:33:01.304971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304822 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-kube-api-access-7hmn5\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-kube-api-access-7hmn5\") pod \"b79597fb-1903-4420-a542-1b036dd4e36c\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " Oct 31 00:33:01.304971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304861 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-kube-api-access-xmh7s\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-kube-api-access-xmh7s\") pod \"18b5e359-4787-45df-b48e-b83fabb5afb9\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " Oct 31 00:33:01.304971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304915 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-certs\") pod \"18b5e359-4787-45df-b48e-b83fabb5afb9\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " Oct 31 00:33:01.305226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.304979 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-socket\") pod \"18b5e359-4787-45df-b48e-b83fabb5afb9\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " Oct 31 00:33:01.305226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305013 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istiod-ca-cert\") pod \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " Oct 31 00:33:01.305226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305027 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-data" pod "fd8769f9-620c-470c-9f4a-c76f5262e9d9" (UID: "fd8769f9-620c-470c-9f4a-c76f5262e9d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.305226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305049 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-credential-socket\") pod \"b79597fb-1903-4420-a542-1b036dd4e36c\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " Oct 31 00:33:01.305226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305096 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-credential-socket\") pod \"18b5e359-4787-45df-b48e-b83fabb5afb9\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " Oct 31 00:33:01.305798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305045 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-credential-socket" pod "fd8769f9-620c-470c-9f4a-c76f5262e9d9" (UID: "fd8769f9-620c-470c-9f4a-c76f5262e9d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.305798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305289 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-socket" pod "18b5e359-4787-45df-b48e-b83fabb5afb9" (UID: "18b5e359-4787-45df-b48e-b83fabb5afb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.305798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305358 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-credential-socket" pod "18b5e359-4787-45df-b48e-b83fabb5afb9" (UID: "18b5e359-4787-45df-b48e-b83fabb5afb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.305798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305466 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b5e359-4787-45df-b48e-b83fabb5afb9-istiod-ca-cert" pod "18b5e359-4787-45df-b48e-b83fabb5afb9" (UID: "18b5e359-4787-45df-b48e-b83fabb5afb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:01.305798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305655 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istiod-ca-cert" pod "fd8769f9-620c-470c-9f4a-c76f5262e9d9" (UID: "fd8769f9-620c-470c-9f4a-c76f5262e9d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:01.305798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305745 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-certs" pod "18b5e359-4787-45df-b48e-b83fabb5afb9" (UID: "18b5e359-4787-45df-b48e-b83fabb5afb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.306405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.305963 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-credential-socket" pod "b79597fb-1903-4420-a542-1b036dd4e36c" (UID: "b79597fb-1903-4420-a542-1b036dd4e36c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.306405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306011 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-data\") pod \"b79597fb-1903-4420-a542-1b036dd4e36c\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " Oct 31 00:33:01.306405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306069 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-podinfo\") pod \"18b5e359-4787-45df-b48e-b83fabb5afb9\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " Oct 31 00:33:01.306405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306222 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-token\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-token\") pod \"18b5e359-4787-45df-b48e-b83fabb5afb9\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " Oct 31 00:33:01.306405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306327 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79597fb-1903-4420-a542-1b036dd4e36c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79597fb-1903-4420-a542-1b036dd4e36c-istiod-ca-cert\") pod \"b79597fb-1903-4420-a542-1b036dd4e36c\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " Oct 31 00:33:01.306405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306381 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-data\") pod \"18b5e359-4787-45df-b48e-b83fabb5afb9\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " Oct 31 00:33:01.306759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306422 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-socket\") pod \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " Oct 31 00:33:01.306759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306494 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-envoy\") pod \"18b5e359-4787-45df-b48e-b83fabb5afb9\" (UID: \"18b5e359-4787-45df-b48e-b83fabb5afb9\") " Oct 31 00:33:01.306759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306531 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-certs\") pod \"b79597fb-1903-4420-a542-1b036dd4e36c\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " Oct 31 00:33:01.306759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306568 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-socket\") pod \"b79597fb-1903-4420-a542-1b036dd4e36c\" (UID: \"b79597fb-1903-4420-a542-1b036dd4e36c\") " Oct 31 00:33:01.306759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306623 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-certs\") pod \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " Oct 31 00:33:01.306759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306686 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-token\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-token\") pod \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " Oct 31 00:33:01.306759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.306737 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-envoy\") pod \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\" (UID: \"fd8769f9-620c-470c-9f4a-c76f5262e9d9\") " Oct 31 00:33:01.307228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307078 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1118a188-069d-4f54-b6a9-fc9cd626ef21-openshift-service-ca\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.307329 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307238 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.307459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307438 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-metrics-certs\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.307527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307491 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vpv\" (UniqueName: \"kubernetes.io/projected/1118a188-069d-4f54-b6a9-fc9cd626ef21-kube-api-access-v4vpv\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.307527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307504 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b79597fb-1903-4420-a542-1b036dd4e36c-istio-podinfo" pod "b79597fb-1903-4420-a542-1b036dd4e36c" (UID: "b79597fb-1903-4420-a542-1b036dd4e36c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:33:01.307665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307552 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-kube-api-access-5d9h8" pod "fd8769f9-620c-470c-9f4a-c76f5262e9d9" (UID: "fd8769f9-620c-470c-9f4a-c76f5262e9d9"). InnerVolumeSpecName "kube-api-access-5d9h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:01.307757 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307731 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-certs" pod "b79597fb-1903-4420-a542-1b036dd4e36c" (UID: "b79597fb-1903-4420-a542-1b036dd4e36c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307801 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307817 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79597fb-1903-4420-a542-1b036dd4e36c-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307819 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-podinfo" pod "fd8769f9-620c-470c-9f4a-c76f5262e9d9" (UID: "fd8769f9-620c-470c-9f4a-c76f5262e9d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307833 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307848 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9h8\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-kube-api-access-5d9h8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307863 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307878 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307892 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6l6j\" (UniqueName: \"kubernetes.io/projected/79da24b1-7783-4b4e-acb0-df82e0165b87-kube-api-access-d6l6j\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307906 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b5e359-4787-45df-b48e-b83fabb5afb9-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307920 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307933 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307946 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307946 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-data" pod "18b5e359-4787-45df-b48e-b83fabb5afb9" (UID: "18b5e359-4787-45df-b48e-b83fabb5afb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307958 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307971 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da24b1-7783-4b4e-acb0-df82e0165b87-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307984 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.307997 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.308002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.308003 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-kube-api-access-xmh7s" pod "18b5e359-4787-45df-b48e-b83fabb5afb9" (UID: "18b5e359-4787-45df-b48e-b83fabb5afb9"). InnerVolumeSpecName "kube-api-access-xmh7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:01.309043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.308024 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79597fb-1903-4420-a542-1b036dd4e36c-istiod-ca-cert" pod "b79597fb-1903-4420-a542-1b036dd4e36c" (UID: "b79597fb-1903-4420-a542-1b036dd4e36c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:01.309043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.308163 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-socket" pod "fd8769f9-620c-470c-9f4a-c76f5262e9d9" (UID: "fd8769f9-620c-470c-9f4a-c76f5262e9d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.309043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.308212 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-envoy" pod "b79597fb-1903-4420-a542-1b036dd4e36c" (UID: "b79597fb-1903-4420-a542-1b036dd4e36c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.309043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.308256 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-socket" pod "b79597fb-1903-4420-a542-1b036dd4e36c" (UID: "b79597fb-1903-4420-a542-1b036dd4e36c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.309043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.308280 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-data" pod "b79597fb-1903-4420-a542-1b036dd4e36c" (UID: "b79597fb-1903-4420-a542-1b036dd4e36c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.309043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.308420 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-kube-api-access-7hmn5" pod "b79597fb-1903-4420-a542-1b036dd4e36c" (UID: "b79597fb-1903-4420-a542-1b036dd4e36c"). InnerVolumeSpecName "kube-api-access-7hmn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:01.309043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.308438 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-certs" pod "fd8769f9-620c-470c-9f4a-c76f5262e9d9" (UID: "fd8769f9-620c-470c-9f4a-c76f5262e9d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.310990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.310955 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-envoy" pod "18b5e359-4787-45df-b48e-b83fabb5afb9" (UID: "18b5e359-4787-45df-b48e-b83fabb5afb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.311214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.311181 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-podinfo" pod "18b5e359-4787-45df-b48e-b83fabb5afb9" (UID: "18b5e359-4787-45df-b48e-b83fabb5afb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:33:01.311532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.311505 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-token" pod "fd8769f9-620c-470c-9f4a-c76f5262e9d9" (UID: "fd8769f9-620c-470c-9f4a-c76f5262e9d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:01.311658 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.311511 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-envoy" pod "fd8769f9-620c-470c-9f4a-c76f5262e9d9" (UID: "fd8769f9-620c-470c-9f4a-c76f5262e9d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:01.312202 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.312173 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-istio-token" pod "b79597fb-1903-4420-a542-1b036dd4e36c" (UID: "b79597fb-1903-4420-a542-1b036dd4e36c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:01.312373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.312348 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-token" pod "18b5e359-4787-45df-b48e-b83fabb5afb9" (UID: "18b5e359-4787-45df-b48e-b83fabb5afb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:01.409397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.409303 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1118a188-069d-4f54-b6a9-fc9cd626ef21-openshift-service-ca\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.409397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.409387 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.409667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.409485 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-metrics-certs\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.409667 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:01.409503 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-serving-cert: secret "maas-api-e2e-mt-f7c92a85-serving-cert" not found Oct 31 00:33:01.409667 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:01.409606 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls podName:1118a188-069d-4f54-b6a9-fc9cd626ef21 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:01.909563072 +0000 UTC m=+1814.384921505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls") pod "maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" (UID: "1118a188-069d-4f54-b6a9-fc9cd626ef21") : secret "maas-api-e2e-mt-f7c92a85-serving-cert" not found Oct 31 00:33:01.409858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.409512 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vpv\" (UniqueName: \"kubernetes.io/projected/1118a188-069d-4f54-b6a9-fc9cd626ef21-kube-api-access-v4vpv\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.409950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.409926 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.409953 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.409968 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.409982 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.409996 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410009 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8769f9-620c-470c-9f4a-c76f5262e9d9-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410022 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410036 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmn5\" (UniqueName: \"kubernetes.io/projected/b79597fb-1903-4420-a542-1b036dd4e36c-kube-api-access-7hmn5\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410051 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmh7s\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-kube-api-access-xmh7s\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410065 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410079 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410091 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410103 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79597fb-1903-4420-a542-1b036dd4e36c-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410117 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410130 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8769f9-620c-470c-9f4a-c76f5262e9d9-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410145 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b5e359-4787-45df-b48e-b83fabb5afb9-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410159 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79597fb-1903-4420-a542-1b036dd4e36c-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:01.410269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.410162 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1118a188-069d-4f54-b6a9-fc9cd626ef21-openshift-service-ca\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.412122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.412095 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-metrics-certs\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.473082 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.473049 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vpv\" (UniqueName: \"kubernetes.io/projected/1118a188-069d-4f54-b6a9-fc9cd626ef21-kube-api-access-v4vpv\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.562896 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.562856 4969 generic.go:356] "Generic (PLEG): container finished" podID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerID="4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8" exitCode=0 Oct 31 00:33:01.563108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.562929 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" Oct 31 00:33:01.563108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.562947 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" event={"ID":"fd8769f9-620c-470c-9f4a-c76f5262e9d9","Type":"ContainerDied","Data":"4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8"} Oct 31 00:33:01.563108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.563005 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6" event={"ID":"fd8769f9-620c-470c-9f4a-c76f5262e9d9","Type":"ContainerDied","Data":"fa197312ef50aff59708d5145bf13d3fd9510a596883d17d5e8f2633f2672e9c"} Oct 31 00:33:01.563108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.563035 4969 scope.go:122] "RemoveContainer" containerID="4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8" Oct 31 00:33:01.571790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.567693 4969 generic.go:356] "Generic (PLEG): container finished" podID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerID="c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471" exitCode=0 Oct 31 00:33:01.571790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.567769 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" event={"ID":"18b5e359-4787-45df-b48e-b83fabb5afb9","Type":"ContainerDied","Data":"c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471"} Oct 31 00:33:01.571790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.567807 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" event={"ID":"18b5e359-4787-45df-b48e-b83fabb5afb9","Type":"ContainerDied","Data":"0f82941aea6d8ddb47ff4f66c5afb68ba487e43b84ed99ad951917b57874c56c"} Oct 31 00:33:01.571790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.567885 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c" Oct 31 00:33:01.573541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.573518 4969 generic.go:356] "Generic (PLEG): container finished" podID="98ee07e7-f923-43eb-b1a4-aec71c96013f" containerID="b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831" exitCode=0 Oct 31 00:33:01.573705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.573579 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" Oct 31 00:33:01.573705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.573660 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" event={"ID":"98ee07e7-f923-43eb-b1a4-aec71c96013f","Type":"ContainerDied","Data":"b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831"} Oct 31 00:33:01.573812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.573705 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6" event={"ID":"98ee07e7-f923-43eb-b1a4-aec71c96013f","Type":"ContainerDied","Data":"cf3ffd11acc97eb9f1cc1424b425eeb1d5c0916ff9951db356a4b2f3a24a761d"} Oct 31 00:33:01.580849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.580812 4969 generic.go:356] "Generic (PLEG): container finished" podID="b79597fb-1903-4420-a542-1b036dd4e36c" containerID="b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc" exitCode=0 Oct 31 00:33:01.580990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.580941 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" event={"ID":"b79597fb-1903-4420-a542-1b036dd4e36c","Type":"ContainerDied","Data":"b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc"} Oct 31 00:33:01.580990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.580977 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" event={"ID":"b79597fb-1903-4420-a542-1b036dd4e36c","Type":"ContainerDied","Data":"a2f3266b209f409e02f95f7d78eb76cd1638486ea520114fe591dc965b81547d"} Oct 31 00:33:01.581105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.581039 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" Oct 31 00:33:01.582831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.582805 4969 generic.go:356] "Generic (PLEG): container finished" podID="79da24b1-7783-4b4e-acb0-df82e0165b87" containerID="12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649" exitCode=0 Oct 31 00:33:01.582947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.582864 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" Oct 31 00:33:01.582947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.582878 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" event={"ID":"79da24b1-7783-4b4e-acb0-df82e0165b87","Type":"ContainerDied","Data":"12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649"} Oct 31 00:33:01.582947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.582928 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt" event={"ID":"79da24b1-7783-4b4e-acb0-df82e0165b87","Type":"ContainerDied","Data":"07da8583b436d371b9ec67866bf7843118e2f48f19ff8a547d808744b58842e2"} Oct 31 00:33:01.584780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.584758 4969 generic.go:356] "Generic (PLEG): container finished" podID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerID="66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8" exitCode=0 Oct 31 00:33:01.584889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.584827 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" event={"ID":"2a8ad439-95c6-4274-8577-4719de10fb3f","Type":"ContainerDied","Data":"66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8"} Oct 31 00:33:01.585391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.585374 4969 scope.go:122] "RemoveContainer" containerID="66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8" Oct 31 00:33:01.586887 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.586863 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:33:01.586963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.586928 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:33:01.586963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.586960 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:33:01.651314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.651291 4969 scope.go:122] "RemoveContainer" containerID="4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8" Oct 31 00:33:01.651741 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:01.651718 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8\": container with ID starting with 4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8 not found: ID does not exist" containerID="4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8" Oct 31 00:33:01.651831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.651753 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8"} err="failed to get container status \"4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8\": rpc error: code = NotFound desc = could not find container \"4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8\": container with ID starting with 4abe8e2ff67d2a2709e03d80bd3964a66cebbee23cca1cb74d5847737d1c55b8 not found: ID does not exist" Oct 31 00:33:01.651831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.651772 4969 scope.go:122] "RemoveContainer" containerID="c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471" Oct 31 00:33:01.670547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.670524 4969 scope.go:122] "RemoveContainer" containerID="ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c" Oct 31 00:33:01.688556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.688538 4969 scope.go:122] "RemoveContainer" containerID="c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471" Oct 31 00:33:01.688961 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:01.688923 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471\": container with ID starting with c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471 not found: ID does not exist" containerID="c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471" Oct 31 00:33:01.689029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.688968 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471"} err="failed to get container status \"c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471\": rpc error: code = NotFound desc = could not find container \"c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471\": container with ID starting with c11e8e6d92507928349cc8f4946c3e3ba700d0ae227080fe34f7e30ac2652471 not found: ID does not exist" Oct 31 00:33:01.689029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.688985 4969 scope.go:122] "RemoveContainer" containerID="ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c" Oct 31 00:33:01.689249 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:01.689230 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c\": container with ID starting with ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c not found: ID does not exist" containerID="ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c" Oct 31 00:33:01.689311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.689260 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c"} err="failed to get container status \"ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c\": rpc error: code = NotFound desc = could not find container \"ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c\": container with ID starting with ac2b95a8540de5a79d4f0251ddc4d95803884dd4c25e9ecda233364c51f2991c not found: ID does not exist" Oct 31 00:33:01.689311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.689283 4969 scope.go:122] "RemoveContainer" containerID="b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831" Oct 31 00:33:01.700487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.700468 4969 scope.go:122] "RemoveContainer" containerID="b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831" Oct 31 00:33:01.700787 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:01.700761 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831\": container with ID starting with b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831 not found: ID does not exist" containerID="b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831" Oct 31 00:33:01.700861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.700784 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831"} err="failed to get container status \"b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831\": rpc error: code = NotFound desc = could not find container \"b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831\": container with ID starting with b48c4c68546822c0efa67da40064c0314e605f93ff5d0bac83b204d765f68831 not found: ID does not exist" Oct 31 00:33:01.700861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.700800 4969 scope.go:122] "RemoveContainer" containerID="b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc" Oct 31 00:33:01.711005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.710987 4969 scope.go:122] "RemoveContainer" containerID="b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc" Oct 31 00:33:01.711289 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:01.711261 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc\": container with ID starting with b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc not found: ID does not exist" containerID="b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc" Oct 31 00:33:01.711375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.711299 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc"} err="failed to get container status \"b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc\": rpc error: code = NotFound desc = could not find container \"b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc\": container with ID starting with b33ba8e7df5d99d24a3f69ee7b76657ec168f6a0f5796aaa3b9cd404c57684fc not found: ID does not exist" Oct 31 00:33:01.711375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.711320 4969 scope.go:122] "RemoveContainer" containerID="12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649" Oct 31 00:33:01.722986 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.722966 4969 scope.go:122] "RemoveContainer" containerID="12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649" Oct 31 00:33:01.723321 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:01.723296 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649\": container with ID starting with 12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649 not found: ID does not exist" containerID="12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649" Oct 31 00:33:01.723434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.723326 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649"} err="failed to get container status \"12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649\": rpc error: code = NotFound desc = could not find container \"12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649\": container with ID starting with 12baafa6ccab0f55cf5e6ce053dbdcf3ecdca4b6047da852629ea490a0907649 not found: ID does not exist" Oct 31 00:33:01.864864 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.864790 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt"] Oct 31 00:33:01.869294 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.869268 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-n9hqt"] Oct 31 00:33:01.918824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.918728 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.921647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.921618 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:01.981482 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.981449 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2"] Oct 31 00:33:01.988428 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:01.988398 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2"] Oct 31 00:33:02.103830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.103785 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:02.179858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.179827 4969 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:33:02.180012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.179882 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-8qmr2" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:33:02.223071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.222993 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6"] Oct 31 00:33:02.230323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.230285 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-28ks6"] Oct 31 00:33:02.328465 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.328434 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6"] Oct 31 00:33:02.333552 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.333522 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-835921-bfccbc495-cjbl6"] Oct 31 00:33:02.442360 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.442276 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c"] Oct 31 00:33:02.446434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.446411 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-crp5c"] Oct 31 00:33:02.503652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.503414 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb"] Oct 31 00:33:02.504170 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:33:02.504133 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1118a188_069d_4f54_b6a9_fc9cd626ef21.slice/crio-bac94ddce7184985b160f16878913ad6e47eaae520f464ff0886c6257e16974b WatchSource:0}: Error finding container bac94ddce7184985b160f16878913ad6e47eaae520f464ff0886c6257e16974b: Status 404 returned error can't find the container with id bac94ddce7184985b160f16878913ad6e47eaae520f464ff0886c6257e16974b Oct 31 00:33:02.615138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.615104 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" event={"ID":"1118a188-069d-4f54-b6a9-fc9cd626ef21","Type":"ContainerStarted","Data":"bac94ddce7184985b160f16878913ad6e47eaae520f464ff0886c6257e16974b"} Oct 31 00:33:02.618023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:02.617996 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" event={"ID":"2a8ad439-95c6-4274-8577-4719de10fb3f","Type":"ContainerStarted","Data":"f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6"} Oct 31 00:33:02.731916 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:02.731831 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:33:03.348781 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.348738 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" podUID="01767835-222a-4076-b9f3-765ad68b4af8" containerName="payload-pre-processing" containerID="cri-o://d83db1c273ef6d2f4317378445dc085b1067d572a3fbf6f3ff7815211b794b94" gracePeriod=25 Oct 31 00:33:03.416761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.416639 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:33:03.418132 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.418100 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:33:03.418258 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.418173 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 00:33:03.643473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.641474 4969 generic.go:356] "Generic (PLEG): container finished" podID="01767835-222a-4076-b9f3-765ad68b4af8" containerID="d83db1c273ef6d2f4317378445dc085b1067d572a3fbf6f3ff7815211b794b94" exitCode=0 Oct 31 00:33:03.644886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.644779 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" event={"ID":"01767835-222a-4076-b9f3-765ad68b4af8","Type":"ContainerDied","Data":"d83db1c273ef6d2f4317378445dc085b1067d572a3fbf6f3ff7815211b794b94"} Oct 31 00:33:03.744585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.744556 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" path="/var/lib/kubelet/pods/18b5e359-4787-45df-b48e-b83fabb5afb9/volumes" Oct 31 00:33:03.745772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.745753 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79da24b1-7783-4b4e-acb0-df82e0165b87" path="/var/lib/kubelet/pods/79da24b1-7783-4b4e-acb0-df82e0165b87/volumes" Oct 31 00:33:03.746420 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.746405 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ee07e7-f923-43eb-b1a4-aec71c96013f" path="/var/lib/kubelet/pods/98ee07e7-f923-43eb-b1a4-aec71c96013f/volumes" Oct 31 00:33:03.747667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.747650 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" path="/var/lib/kubelet/pods/b79597fb-1903-4420-a542-1b036dd4e36c/volumes" Oct 31 00:33:03.748539 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.748522 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" path="/var/lib/kubelet/pods/fd8769f9-620c-470c-9f4a-c76f5262e9d9/volumes" Oct 31 00:33:03.808359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.808318 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:33:03.869571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.869535 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01767835-222a-4076-b9f3-765ad68b4af8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01767835-222a-4076-b9f3-765ad68b4af8-plugins-config-volume\") pod \"01767835-222a-4076-b9f3-765ad68b4af8\" (UID: \"01767835-222a-4076-b9f3-765ad68b4af8\") " Oct 31 00:33:03.869826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.869632 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01767835-222a-4076-b9f3-765ad68b4af8-kube-api-access-5sstj\" (UniqueName: \"kubernetes.io/projected/01767835-222a-4076-b9f3-765ad68b4af8-kube-api-access-5sstj\") pod \"01767835-222a-4076-b9f3-765ad68b4af8\" (UID: \"01767835-222a-4076-b9f3-765ad68b4af8\") " Oct 31 00:33:03.871041 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.871015 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01767835-222a-4076-b9f3-765ad68b4af8-plugins-config-volume" pod "01767835-222a-4076-b9f3-765ad68b4af8" (UID: "01767835-222a-4076-b9f3-765ad68b4af8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:03.874930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.874893 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01767835-222a-4076-b9f3-765ad68b4af8-kube-api-access-5sstj" pod "01767835-222a-4076-b9f3-765ad68b4af8" (UID: "01767835-222a-4076-b9f3-765ad68b4af8"). InnerVolumeSpecName "kube-api-access-5sstj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:03.971404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.971168 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01767835-222a-4076-b9f3-765ad68b4af8-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:03.971404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:03.971192 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sstj\" (UniqueName: \"kubernetes.io/projected/01767835-222a-4076-b9f3-765ad68b4af8-kube-api-access-5sstj\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:04.417309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:04.417266 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:33:04.417478 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:04.417337 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 00:33:04.658377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:04.658346 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" Oct 31 00:33:04.658377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:04.658362 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7" event={"ID":"01767835-222a-4076-b9f3-765ad68b4af8","Type":"ContainerDied","Data":"c71915c36c49875570840a69a9ae736c69040b955f60c97b3c57ca35dc64398e"} Oct 31 00:33:04.658860 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:04.658431 4969 scope.go:122] "RemoveContainer" containerID="d83db1c273ef6d2f4317378445dc085b1067d572a3fbf6f3ff7815211b794b94" Oct 31 00:33:04.829477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:04.829434 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7"] Oct 31 00:33:04.875966 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:04.875934 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-5cbb9b5ccf-lndr7"] Oct 31 00:33:05.057464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:05.057383 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:33:05.416461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:05.416423 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:33:05.416674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:05.416489 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 00:33:05.739909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:05.739827 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01767835-222a-4076-b9f3-765ad68b4af8" path="/var/lib/kubelet/pods/01767835-222a-4076-b9f3-765ad68b4af8/volumes" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.174756 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv"] Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175480 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175494 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175518 4969 cpu_manager.go:414] "containerMap: removing container" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175524 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175542 4969 cpu_manager.go:414] "containerMap: removing container" podUID="01767835-222a-4076-b9f3-765ad68b4af8" containerName="payload-pre-processing" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175551 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01767835-222a-4076-b9f3-765ad68b4af8" containerName="payload-pre-processing" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175559 4969 cpu_manager.go:414] "containerMap: removing container" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175564 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175720 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79597fb-1903-4420-a542-1b036dd4e36c" containerName="istio-proxy" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175735 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175745 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="01767835-222a-4076-b9f3-765ad68b4af8" containerName="payload-pre-processing" Oct 31 00:33:06.179509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.175757 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8769f9-620c-470c-9f4a-c76f5262e9d9" containerName="istio-proxy" Oct 31 00:33:06.216835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.216018 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:06.244521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.244480 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv"] Oct 31 00:33:06.300990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.300953 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-70babe\"" Oct 31 00:33:06.309621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.301438 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-70babe-dockercfg-7pcsq\"" Oct 31 00:33:06.347958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.346715 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dab119b-ecff-4560-a864-880e3b8c87da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv\" (UID: \"3dab119b-ecff-4560-a864-880e3b8c87da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:06.347958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.346876 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27pv\" (UniqueName: \"kubernetes.io/projected/3dab119b-ecff-4560-a864-880e3b8c87da-kube-api-access-g27pv\") pod \"payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv\" (UID: \"3dab119b-ecff-4560-a864-880e3b8c87da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:06.366616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.359936 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds"] Oct 31 00:33:06.366616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.362077 4969 cpu_manager.go:414] "containerMap: removing container" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" Oct 31 00:33:06.366616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.362099 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" Oct 31 00:33:06.366616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.362333 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b5e359-4787-45df-b48e-b83fabb5afb9" containerName="istio-proxy" Oct 31 00:33:06.370812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.369914 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:06.407647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.382560 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds"] Oct 31 00:33:06.429690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.417164 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:33:06.429690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.417241 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 00:33:06.453429 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.449286 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27pv\" (UniqueName: \"kubernetes.io/projected/3dab119b-ecff-4560-a864-880e3b8c87da-kube-api-access-g27pv\") pod \"payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv\" (UID: \"3dab119b-ecff-4560-a864-880e3b8c87da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:06.453429 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.449618 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dab119b-ecff-4560-a864-880e3b8c87da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv\" (UID: \"3dab119b-ecff-4560-a864-880e3b8c87da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:06.453429 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.451059 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dab119b-ecff-4560-a864-880e3b8c87da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv\" (UID: \"3dab119b-ecff-4560-a864-880e3b8c87da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:06.542977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.542941 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27pv\" (UniqueName: \"kubernetes.io/projected/3dab119b-ecff-4560-a864-880e3b8c87da-kube-api-access-g27pv\") pod \"payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv\" (UID: \"3dab119b-ecff-4560-a864-880e3b8c87da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:06.551303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.551042 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqjw\" (UniqueName: \"kubernetes.io/projected/74a85cff-6b8c-41de-b5dc-31b0e048dd26-kube-api-access-9dqjw\") pod \"payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds\" (UID: \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:06.551303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.551112 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a85cff-6b8c-41de-b5dc-31b0e048dd26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds\" (UID: \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:06.551303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.551225 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:06.654959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.654308 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqjw\" (UniqueName: \"kubernetes.io/projected/74a85cff-6b8c-41de-b5dc-31b0e048dd26-kube-api-access-9dqjw\") pod \"payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds\" (UID: \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:06.654959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.654381 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a85cff-6b8c-41de-b5dc-31b0e048dd26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds\" (UID: \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:06.655567 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.655532 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a85cff-6b8c-41de-b5dc-31b0e048dd26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds\" (UID: \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:06.721134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.718748 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqjw\" (UniqueName: \"kubernetes.io/projected/74a85cff-6b8c-41de-b5dc-31b0e048dd26-kube-api-access-9dqjw\") pod \"payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds\" (UID: \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:06.758231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:06.758201 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:07.018614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.014260 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv"] Oct 31 00:33:07.025122 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:33:07.025083 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dab119b_ecff_4560_a864_880e3b8c87da.slice/crio-112737d7c4f24a9b003293c8f63fdac455142ea0a902ac1980b65a5b93741508 WatchSource:0}: Error finding container 112737d7c4f24a9b003293c8f63fdac455142ea0a902ac1980b65a5b93741508: Status 404 returned error can't find the container with id 112737d7c4f24a9b003293c8f63fdac455142ea0a902ac1980b65a5b93741508 Oct 31 00:33:07.211011 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:33:07.210974 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a85cff_6b8c_41de_b5dc_31b0e048dd26.slice/crio-3bb3242b076ddb69e402db648d2a3faa5d1de21871bbfe49f70ccb3bb92bd242 WatchSource:0}: Error finding container 3bb3242b076ddb69e402db648d2a3faa5d1de21871bbfe49f70ccb3bb92bd242: Status 404 returned error can't find the container with id 3bb3242b076ddb69e402db648d2a3faa5d1de21871bbfe49f70ccb3bb92bd242 Oct 31 00:33:07.211266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.211239 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds"] Oct 31 00:33:07.417047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.417008 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 00:33:07.417229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.417081 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 00:33:07.726253 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.726221 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" event={"ID":"74a85cff-6b8c-41de-b5dc-31b0e048dd26","Type":"ContainerStarted","Data":"1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079"} Oct 31 00:33:07.726454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.726292 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" event={"ID":"74a85cff-6b8c-41de-b5dc-31b0e048dd26","Type":"ContainerStarted","Data":"3bb3242b076ddb69e402db648d2a3faa5d1de21871bbfe49f70ccb3bb92bd242"} Oct 31 00:33:07.741013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.740979 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" event={"ID":"3dab119b-ecff-4560-a864-880e3b8c87da","Type":"ContainerStarted","Data":"2d7da94deb0fd2e39da7fc06227652d2acbea750f21cca65af5c9b1c85bf4c4e"} Oct 31 00:33:07.741144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.741039 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:07.741144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.741058 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:07.741144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.741072 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:07.741144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.741086 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" event={"ID":"3dab119b-ecff-4560-a864-880e3b8c87da","Type":"ContainerStarted","Data":"112737d7c4f24a9b003293c8f63fdac455142ea0a902ac1980b65a5b93741508"} Oct 31 00:33:07.741144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.741099 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" event={"ID":"1118a188-069d-4f54-b6a9-fc9cd626ef21","Type":"ContainerStarted","Data":"66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513"} Oct 31 00:33:07.866973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:07.866927 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" podStartSLOduration=1.866913173 podStartE2EDuration="1.866913173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:07.864282543 +0000 UTC m=+1820.339640966" watchObservedRunningTime="2026-10-31 00:33:07.866913173 +0000 UTC m=+1820.342271614" Oct 31 00:33:08.001338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:08.001276 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" podStartSLOduration=2.001257427 podStartE2EDuration="2.001257427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:07.997915684 +0000 UTC m=+1820.473274108" watchObservedRunningTime="2026-10-31 00:33:08.001257427 +0000 UTC m=+1820.476615849" Oct 31 00:33:08.122931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:08.122792 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" podStartSLOduration=2.191688037 podStartE2EDuration="7.122755395s" totalImagesPullingTime="4.931067358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:08.115500527 +0000 UTC m=+1820.590858950" watchObservedRunningTime="2026-10-31 00:33:08.122755395 +0000 UTC m=+1820.598113858" Oct 31 00:33:08.421206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:08.421129 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:33:08.422261 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:08.422238 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:33:13.547378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:13.547346 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx"] Oct 31 00:33:13.596408 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:13.596376 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4dc2e-knnfx"] Oct 31 00:33:13.736184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:13.736153 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e451a9be-f1e8-4fcb-b911-15beb654c27b" path="/var/lib/kubelet/pods/e451a9be-f1e8-4fcb-b911-15beb654c27b/volumes" Oct 31 00:33:14.062163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:14.062122 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg"] Oct 31 00:33:14.111539 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:14.111510 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c916d419-mllrg"] Oct 31 00:33:14.762341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:14.762309 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:33:15.736561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:15.736530 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69269ffe-3005-4849-abd1-97462a39286f" path="/var/lib/kubelet/pods/69269ffe-3005-4849-abd1-97462a39286f/volumes" Oct 31 00:33:18.751652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:18.751581 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:33:19.759160 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:19.759131 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:33:20.459195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:20.459157 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m"] Oct 31 00:33:20.459394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:20.459369 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" containerID="cri-o://19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4" gracePeriod=30 Oct 31 00:33:25.814243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.814210 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:33:25.909743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.909711 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-certs\") pod \"439b8775-aa5e-4faa-b117-08be2dfe0d00\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " Oct 31 00:33:25.909913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.909776 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-data\") pod \"439b8775-aa5e-4faa-b117-08be2dfe0d00\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " Oct 31 00:33:25.909913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.909802 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-socket\") pod \"439b8775-aa5e-4faa-b117-08be2dfe0d00\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " Oct 31 00:33:25.909913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.909898 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439b8775-aa5e-4faa-b117-08be2dfe0d00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439b8775-aa5e-4faa-b117-08be2dfe0d00-istiod-ca-cert\") pod \"439b8775-aa5e-4faa-b117-08be2dfe0d00\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " Oct 31 00:33:25.910074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.909924 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-token\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-token\") pod \"439b8775-aa5e-4faa-b117-08be2dfe0d00\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " Oct 31 00:33:25.910074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.909964 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-envoy\") pod \"439b8775-aa5e-4faa-b117-08be2dfe0d00\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " Oct 31 00:33:25.910074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.909997 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-certs" pod "439b8775-aa5e-4faa-b117-08be2dfe0d00" (UID: "439b8775-aa5e-4faa-b117-08be2dfe0d00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:25.910074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.910015 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-credential-socket\") pod \"439b8775-aa5e-4faa-b117-08be2dfe0d00\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " Oct 31 00:33:25.910227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.910080 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-data" pod "439b8775-aa5e-4faa-b117-08be2dfe0d00" (UID: "439b8775-aa5e-4faa-b117-08be2dfe0d00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:25.910227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.910178 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-socket" pod "439b8775-aa5e-4faa-b117-08be2dfe0d00" (UID: "439b8775-aa5e-4faa-b117-08be2dfe0d00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:25.910341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.910233 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-podinfo\") pod \"439b8775-aa5e-4faa-b117-08be2dfe0d00\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " Oct 31 00:33:25.910341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.910293 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-kube-api-access-kp2c8\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-kube-api-access-kp2c8\") pod \"439b8775-aa5e-4faa-b117-08be2dfe0d00\" (UID: \"439b8775-aa5e-4faa-b117-08be2dfe0d00\") " Oct 31 00:33:25.910909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.910713 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439b8775-aa5e-4faa-b117-08be2dfe0d00-istiod-ca-cert" pod "439b8775-aa5e-4faa-b117-08be2dfe0d00" (UID: "439b8775-aa5e-4faa-b117-08be2dfe0d00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:25.910909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.910768 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-credential-socket" pod "439b8775-aa5e-4faa-b117-08be2dfe0d00" (UID: "439b8775-aa5e-4faa-b117-08be2dfe0d00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:25.911183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.911161 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:25.911277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.911188 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:25.911277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.911202 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:25.911277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.911215 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:25.911277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.911228 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439b8775-aa5e-4faa-b117-08be2dfe0d00-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:25.912994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.912964 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-podinfo" pod "439b8775-aa5e-4faa-b117-08be2dfe0d00" (UID: "439b8775-aa5e-4faa-b117-08be2dfe0d00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:33:25.913327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.913304 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-token" pod "439b8775-aa5e-4faa-b117-08be2dfe0d00" (UID: "439b8775-aa5e-4faa-b117-08be2dfe0d00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:25.913762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.913743 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-kube-api-access-kp2c8" pod "439b8775-aa5e-4faa-b117-08be2dfe0d00" (UID: "439b8775-aa5e-4faa-b117-08be2dfe0d00"). InnerVolumeSpecName "kube-api-access-kp2c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:25.913896 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.913872 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-envoy" pod "439b8775-aa5e-4faa-b117-08be2dfe0d00" (UID: "439b8775-aa5e-4faa-b117-08be2dfe0d00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:25.923785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.923752 4969 generic.go:356] "Generic (PLEG): container finished" podID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerID="19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4" exitCode=0 Oct 31 00:33:25.923996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.923981 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" event={"ID":"439b8775-aa5e-4faa-b117-08be2dfe0d00","Type":"ContainerDied","Data":"19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4"} Oct 31 00:33:25.924160 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.924024 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" event={"ID":"439b8775-aa5e-4faa-b117-08be2dfe0d00","Type":"ContainerDied","Data":"4f6da55602aa10fef9ad0183a6553fe04c5d43918d2460666d098d277c45caba"} Oct 31 00:33:25.924160 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.924044 4969 scope.go:122] "RemoveContainer" containerID="19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4" Oct 31 00:33:25.924232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.924165 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m" Oct 31 00:33:25.947227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.947205 4969 scope.go:122] "RemoveContainer" containerID="72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a" Oct 31 00:33:25.961004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.960985 4969 scope.go:122] "RemoveContainer" containerID="19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4" Oct 31 00:33:25.961295 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:25.961279 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4\": container with ID starting with 19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4 not found: ID does not exist" containerID="19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4" Oct 31 00:33:25.961333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.961304 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4"} err="failed to get container status \"19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4\": rpc error: code = NotFound desc = could not find container \"19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4\": container with ID starting with 19fe09c04bddfc69c0a14bdfbae17be07e35736783d509bb8ec2663556f29ef4 not found: ID does not exist" Oct 31 00:33:25.961333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.961320 4969 scope.go:122] "RemoveContainer" containerID="72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a" Oct 31 00:33:25.961578 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:25.961530 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a\": container with ID starting with 72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a not found: ID does not exist" containerID="72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a" Oct 31 00:33:25.961578 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:25.961557 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a"} err="failed to get container status \"72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a\": rpc error: code = NotFound desc = could not find container \"72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a\": container with ID starting with 72ed63283dcbad908c63ea79518e1decb63153e473322c781b8a06db90ed8f4a not found: ID does not exist" Oct 31 00:33:26.013097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:26.012874 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:26.013097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:26.012899 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:26.013097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:26.012913 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2c8\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-kube-api-access-kp2c8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:26.013097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:26.012923 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439b8775-aa5e-4faa-b117-08be2dfe0d00-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:26.109201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:26.109090 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m"] Oct 31 00:33:26.116077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:26.116050 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a4dc2e-openshift-default-74d88b7bf9-rj76m"] Oct 31 00:33:27.738363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:27.738331 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" path="/var/lib/kubelet/pods/439b8775-aa5e-4faa-b117-08be2dfe0d00/volumes" Oct 31 00:33:32.728874 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:32.728833 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:33:32.794774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:32.794734 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq"] Oct 31 00:33:33.770783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.770748 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg"] Oct 31 00:33:33.771486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.771471 4969 cpu_manager.go:414] "containerMap: removing container" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" Oct 31 00:33:33.771530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.771487 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" Oct 31 00:33:33.771530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.771511 4969 cpu_manager.go:414] "containerMap: removing container" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" Oct 31 00:33:33.771530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.771516 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" Oct 31 00:33:33.771670 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.771659 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" Oct 31 00:33:33.771703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.771673 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="439b8775-aa5e-4faa-b117-08be2dfe0d00" containerName="istio-proxy" Oct 31 00:33:33.776778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.776759 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.818846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.818815 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.818985 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.818868 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-metrics-certs\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.818985 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.818935 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deed1d-3a2c-40c7-bd4d-f171cc753481-openshift-service-ca\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.819064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.819040 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjcx\" (UniqueName: \"kubernetes.io/projected/94deed1d-3a2c-40c7-bd4d-f171cc753481-kube-api-access-zkjcx\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.828930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.828905 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2cda64-serving-cert\"" Oct 31 00:33:33.833430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.833402 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg"] Oct 31 00:33:33.921026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.920991 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.921209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.921057 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-metrics-certs\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.921209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.921086 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deed1d-3a2c-40c7-bd4d-f171cc753481-openshift-service-ca\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.921209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.921150 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjcx\" (UniqueName: \"kubernetes.io/projected/94deed1d-3a2c-40c7-bd4d-f171cc753481-kube-api-access-zkjcx\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.921570 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:33.921554 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-serving-cert: secret "maas-api-e2e-api-b-2cda64-serving-cert" not found Oct 31 00:33:33.921678 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:33.921665 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls podName:94deed1d-3a2c-40c7-bd4d-f171cc753481 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:34.421643724 +0000 UTC m=+1846.897002131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls") pod "maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" (UID: "94deed1d-3a2c-40c7-bd4d-f171cc753481") : secret "maas-api-e2e-api-b-2cda64-serving-cert" not found Oct 31 00:33:33.922530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.922478 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deed1d-3a2c-40c7-bd4d-f171cc753481-openshift-service-ca\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.924456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.924432 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-metrics-certs\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:33.984352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:33.984324 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjcx\" (UniqueName: \"kubernetes.io/projected/94deed1d-3a2c-40c7-bd4d-f171cc753481-kube-api-access-zkjcx\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:34.434923 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:34.434878 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:34.438277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:34.438248 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls\") pod \"maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:34.691915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:34.691832 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:35.078888 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:33:35.078859 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94deed1d_3a2c_40c7_bd4d_f171cc753481.slice/crio-238f2200611e09b133aa97ddf38e6270082c82f7af9865b04c76d15575947509 WatchSource:0}: Error finding container 238f2200611e09b133aa97ddf38e6270082c82f7af9865b04c76d15575947509: Status 404 returned error can't find the container with id 238f2200611e09b133aa97ddf38e6270082c82f7af9865b04c76d15575947509 Oct 31 00:33:35.079377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:35.079353 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg"] Oct 31 00:33:35.896053 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:35.896018 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp"] Oct 31 00:33:35.896253 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:35.896229 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" containerID="cri-o://8c70bf0ce2b9a1426fdb8a1f14262574890e97f27183aa56c33e33d165524d0a" gracePeriod=30 Oct 31 00:33:36.013683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:36.013648 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" event={"ID":"94deed1d-3a2c-40c7-bd4d-f171cc753481","Type":"ContainerStarted","Data":"238f2200611e09b133aa97ddf38e6270082c82f7af9865b04c76d15575947509"} Oct 31 00:33:37.031084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:37.031042 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" event={"ID":"94deed1d-3a2c-40c7-bd4d-f171cc753481","Type":"ContainerStarted","Data":"2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8"} Oct 31 00:33:37.032937 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:37.032912 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:37.151537 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:37.151457 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" podStartSLOduration=3.306943621 podStartE2EDuration="4.151441758s" totalImagesPullingTime="844.498137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:37.14982942 +0000 UTC m=+1849.625187844" watchObservedRunningTime="2026-10-31 00:33:37.151441758 +0000 UTC m=+1849.626800180" Oct 31 00:33:37.795517 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:37.795468 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" podUID="29465fcc-6627-4e1b-96db-5c9f28b9f57e" containerName="payload-processing" containerID="cri-o://c4065fa63d2f418b4f01942951286656a8b739773fdc264805dd3253fe94af46" gracePeriod=25 Oct 31 00:33:38.044636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:38.044584 4969 generic.go:356] "Generic (PLEG): container finished" podID="29465fcc-6627-4e1b-96db-5c9f28b9f57e" containerID="c4065fa63d2f418b4f01942951286656a8b739773fdc264805dd3253fe94af46" exitCode=0 Oct 31 00:33:38.045506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:38.045483 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" event={"ID":"29465fcc-6627-4e1b-96db-5c9f28b9f57e","Type":"ContainerDied","Data":"c4065fa63d2f418b4f01942951286656a8b739773fdc264805dd3253fe94af46"} Oct 31 00:33:38.094888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:38.094862 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:33:38.188795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:38.188761 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29465fcc-6627-4e1b-96db-5c9f28b9f57e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29465fcc-6627-4e1b-96db-5c9f28b9f57e-plugins-config-volume\") pod \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\" (UID: \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\") " Oct 31 00:33:38.188969 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:38.188932 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29465fcc-6627-4e1b-96db-5c9f28b9f57e-kube-api-access-b8xqd\" (UniqueName: \"kubernetes.io/projected/29465fcc-6627-4e1b-96db-5c9f28b9f57e-kube-api-access-b8xqd\") pod \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\" (UID: \"29465fcc-6627-4e1b-96db-5c9f28b9f57e\") " Oct 31 00:33:38.189228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:38.189198 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29465fcc-6627-4e1b-96db-5c9f28b9f57e-plugins-config-volume" pod "29465fcc-6627-4e1b-96db-5c9f28b9f57e" (UID: "29465fcc-6627-4e1b-96db-5c9f28b9f57e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:38.189585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:38.189564 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29465fcc-6627-4e1b-96db-5c9f28b9f57e-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:38.191213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:38.191187 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29465fcc-6627-4e1b-96db-5c9f28b9f57e-kube-api-access-b8xqd" pod "29465fcc-6627-4e1b-96db-5c9f28b9f57e" (UID: "29465fcc-6627-4e1b-96db-5c9f28b9f57e"). InnerVolumeSpecName "kube-api-access-b8xqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:38.290639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:38.290582 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8xqd\" (UniqueName: \"kubernetes.io/projected/29465fcc-6627-4e1b-96db-5c9f28b9f57e-kube-api-access-b8xqd\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:39.055141 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:39.055108 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" Oct 31 00:33:39.055578 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:39.055145 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq" event={"ID":"29465fcc-6627-4e1b-96db-5c9f28b9f57e","Type":"ContainerDied","Data":"ab43b884dfb3c7d7125eeb77768aa09c6883785974e5b4e5da574024ccbd6f8d"} Oct 31 00:33:39.055578 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:39.055215 4969 scope.go:122] "RemoveContainer" containerID="c4065fa63d2f418b4f01942951286656a8b739773fdc264805dd3253fe94af46" Oct 31 00:33:39.221809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:39.221769 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq"] Oct 31 00:33:39.270849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:39.270816 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ccdde-8595f9fdd7-vprsq"] Oct 31 00:33:39.737223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:39.737191 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29465fcc-6627-4e1b-96db-5c9f28b9f57e" path="/var/lib/kubelet/pods/29465fcc-6627-4e1b-96db-5c9f28b9f57e/volumes" Oct 31 00:33:40.984252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:40.984217 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 00:33:40.984649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:40.984283 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 00:33:41.089234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.089199 4969 generic.go:356] "Generic (PLEG): container finished" podID="dc817050-32a6-4b90-9a36-58c5be3da104" containerID="8c70bf0ce2b9a1426fdb8a1f14262574890e97f27183aa56c33e33d165524d0a" exitCode=0 Oct 31 00:33:41.089365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.089269 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" event={"ID":"dc817050-32a6-4b90-9a36-58c5be3da104","Type":"ContainerDied","Data":"8c70bf0ce2b9a1426fdb8a1f14262574890e97f27183aa56c33e33d165524d0a"} Oct 31 00:33:41.182529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.182504 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:33:41.330874 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.330821 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-certs\") pod \"dc817050-32a6-4b90-9a36-58c5be3da104\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " Oct 31 00:33:41.331074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.330931 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-credential-socket\") pod \"dc817050-32a6-4b90-9a36-58c5be3da104\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " Oct 31 00:33:41.331074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.331058 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc817050-32a6-4b90-9a36-58c5be3da104-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc817050-32a6-4b90-9a36-58c5be3da104-istiod-ca-cert\") pod \"dc817050-32a6-4b90-9a36-58c5be3da104\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " Oct 31 00:33:41.331209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.331090 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-envoy\") pod \"dc817050-32a6-4b90-9a36-58c5be3da104\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " Oct 31 00:33:41.331209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.331097 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-certs" pod "dc817050-32a6-4b90-9a36-58c5be3da104" (UID: "dc817050-32a6-4b90-9a36-58c5be3da104"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:41.331209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.331119 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-kube-api-access-bg62h\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-kube-api-access-bg62h\") pod \"dc817050-32a6-4b90-9a36-58c5be3da104\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " Oct 31 00:33:41.331209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.331139 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-istio-token\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-istio-token\") pod \"dc817050-32a6-4b90-9a36-58c5be3da104\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " Oct 31 00:33:41.331209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.331179 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc817050-32a6-4b90-9a36-58c5be3da104-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc817050-32a6-4b90-9a36-58c5be3da104-istio-podinfo\") pod \"dc817050-32a6-4b90-9a36-58c5be3da104\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " Oct 31 00:33:41.331499 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.331223 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-data\") pod \"dc817050-32a6-4b90-9a36-58c5be3da104\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " Oct 31 00:33:41.331499 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.331251 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-socket\") pod \"dc817050-32a6-4b90-9a36-58c5be3da104\" (UID: \"dc817050-32a6-4b90-9a36-58c5be3da104\") " Oct 31 00:33:41.332066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.331841 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc817050-32a6-4b90-9a36-58c5be3da104-istiod-ca-cert" pod "dc817050-32a6-4b90-9a36-58c5be3da104" (UID: "dc817050-32a6-4b90-9a36-58c5be3da104"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:41.332197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.332087 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:41.332355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.332326 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-credential-socket" pod "dc817050-32a6-4b90-9a36-58c5be3da104" (UID: "dc817050-32a6-4b90-9a36-58c5be3da104"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:41.332876 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.332667 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-data" pod "dc817050-32a6-4b90-9a36-58c5be3da104" (UID: "dc817050-32a6-4b90-9a36-58c5be3da104"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:41.333160 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.333081 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-socket" pod "dc817050-32a6-4b90-9a36-58c5be3da104" (UID: "dc817050-32a6-4b90-9a36-58c5be3da104"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:41.335046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.335002 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-envoy" pod "dc817050-32a6-4b90-9a36-58c5be3da104" (UID: "dc817050-32a6-4b90-9a36-58c5be3da104"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:41.337026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.336999 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-kube-api-access-bg62h" pod "dc817050-32a6-4b90-9a36-58c5be3da104" (UID: "dc817050-32a6-4b90-9a36-58c5be3da104"). InnerVolumeSpecName "kube-api-access-bg62h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:41.337118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.337025 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-istio-token" pod "dc817050-32a6-4b90-9a36-58c5be3da104" (UID: "dc817050-32a6-4b90-9a36-58c5be3da104"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:41.337433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.337413 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc817050-32a6-4b90-9a36-58c5be3da104-istio-podinfo" pod "dc817050-32a6-4b90-9a36-58c5be3da104" (UID: "dc817050-32a6-4b90-9a36-58c5be3da104"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:33:41.433493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.433413 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc817050-32a6-4b90-9a36-58c5be3da104-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:41.433493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.433443 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:41.433493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.433453 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg62h\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-kube-api-access-bg62h\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:41.433493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.433462 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc817050-32a6-4b90-9a36-58c5be3da104-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:41.433493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.433470 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc817050-32a6-4b90-9a36-58c5be3da104-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:41.433493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.433479 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:41.433493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.433488 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:41.433493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:41.433497 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc817050-32a6-4b90-9a36-58c5be3da104-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:42.099375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:42.099279 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" Oct 31 00:33:42.099375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:42.099317 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp" event={"ID":"dc817050-32a6-4b90-9a36-58c5be3da104","Type":"ContainerDied","Data":"2d71ae8e1bd4b557e02cc4f91c6a815f7b983c1f85081f96829f22572770578c"} Oct 31 00:33:42.099904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:42.099396 4969 scope.go:122] "RemoveContainer" containerID="8c70bf0ce2b9a1426fdb8a1f14262574890e97f27183aa56c33e33d165524d0a" Oct 31 00:33:42.271667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:42.271633 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp"] Oct 31 00:33:42.321025 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:42.320988 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-5tndp"] Oct 31 00:33:43.735939 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:43.735903 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" path="/var/lib/kubelet/pods/dc817050-32a6-4b90-9a36-58c5be3da104/volumes" Oct 31 00:33:44.061388 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:44.061316 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:33:45.789910 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.789879 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74"] Oct 31 00:33:45.790547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.790532 4969 cpu_manager.go:414] "containerMap: removing container" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" Oct 31 00:33:45.790547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.790547 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" Oct 31 00:33:45.790643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.790572 4969 cpu_manager.go:414] "containerMap: removing container" podUID="29465fcc-6627-4e1b-96db-5c9f28b9f57e" containerName="payload-processing" Oct 31 00:33:45.790643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.790577 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29465fcc-6627-4e1b-96db-5c9f28b9f57e" containerName="payload-processing" Oct 31 00:33:45.790718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.790708 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc817050-32a6-4b90-9a36-58c5be3da104" containerName="istio-proxy" Oct 31 00:33:45.790753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.790725 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="29465fcc-6627-4e1b-96db-5c9f28b9f57e" containerName="payload-processing" Oct 31 00:33:45.795314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.795292 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.896390 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.896344 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.896587 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.896424 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsgqh\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-kube-api-access-fsgqh\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.896587 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.896562 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.897170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.897143 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.897359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.897269 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.897359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.897321 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.897543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.897399 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.897543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.897457 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.897543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.897512 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.904327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.904302 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74"] Oct 31 00:33:45.998699 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.998666 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.998885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.998730 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.998885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.998785 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsgqh\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-kube-api-access-fsgqh\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.998885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.998837 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.998901 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.998987 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.999031 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.999095 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.999141 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.999276 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.999304 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.999480 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.999543 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:45.999853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:45.999833 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:46.001483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:46.001460 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:46.001586 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:46.001492 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:46.069361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:46.069286 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:46.069361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:46.069343 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsgqh\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-kube-api-access-fsgqh\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:46.109830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:46.109792 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:46.567110 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:33:46.567078 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025a7d0c_7431_4b83_961e_46dceb7ebc1e.slice/crio-cac5b1b1ad4d09117be09bf50ea4bac03005580f6aaa75525918b3b2142bc616 WatchSource:0}: Error finding container cac5b1b1ad4d09117be09bf50ea4bac03005580f6aaa75525918b3b2142bc616: Status 404 returned error can't find the container with id cac5b1b1ad4d09117be09bf50ea4bac03005580f6aaa75525918b3b2142bc616 Oct 31 00:33:46.567690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:46.567581 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74"] Oct 31 00:33:46.569729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:46.569701 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:33:46.569811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:46.569760 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:33:46.569811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:46.569788 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:33:47.145297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:47.145255 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" event={"ID":"025a7d0c-7431-4b83-961e-46dceb7ebc1e","Type":"ContainerStarted","Data":"40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9"} Oct 31 00:33:47.145858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:47.145317 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" event={"ID":"025a7d0c-7431-4b83-961e-46dceb7ebc1e","Type":"ContainerStarted","Data":"cac5b1b1ad4d09117be09bf50ea4bac03005580f6aaa75525918b3b2142bc616"} Oct 31 00:33:47.314662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:47.314610 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" podStartSLOduration=2.314582784 podStartE2EDuration="2.314582784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:47.31025616 +0000 UTC m=+1859.785614593" watchObservedRunningTime="2026-10-31 00:33:47.314582784 +0000 UTC m=+1859.789941206" Oct 31 00:33:48.110197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:48.110160 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:48.112050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:48.112022 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:33:48.112184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:48.112085 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 31 00:33:48.707779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:48.707749 4969 scope.go:122] "RemoveContainer" containerID="a935867082154c5cdbb6baa4c21e9035b64d2c3172fe1b4b9775191a158c56cc" Oct 31 00:33:48.721739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:48.721666 4969 scope.go:122] "RemoveContainer" containerID="791ec20ac4b24f69d5e6edf42701e4ade253ad89e68d4e7e0dfc7de14700120e" Oct 31 00:33:49.110880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:49.110801 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:33:49.110880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:49.110869 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 31 00:33:50.110429 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:50.110389 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:33:50.110888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:50.110450 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 31 00:33:51.024713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:51.024676 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9"] Oct 31 00:33:51.024910 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:51.024885 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" containerID="cri-o://764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898" gracePeriod=30 Oct 31 00:33:51.111210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:51.111173 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:33:51.111653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:51.111236 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 31 00:33:52.110365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:52.110332 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 31 00:33:52.110553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:52.110389 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 31 00:33:53.116720 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:53.116682 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:53.117320 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:53.117304 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:53.118004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:53.117974 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:33:54.608007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.607976 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l"] Oct 31 00:33:54.614494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.614475 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:33:54.669536 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.669499 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l"] Oct 31 00:33:54.733773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.720934 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bba5ed\"" Oct 31 00:33:54.760930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.735611 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bba5ed-dockercfg-strgd\"" Oct 31 00:33:54.760930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.759386 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkj8\" (UniqueName: \"kubernetes.io/projected/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-kube-api-access-hrkj8\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l\" (UID: \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:33:54.760930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.759511 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l\" (UID: \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:33:54.861431 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.861331 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkj8\" (UniqueName: \"kubernetes.io/projected/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-kube-api-access-hrkj8\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l\" (UID: \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:33:54.861627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.861520 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l\" (UID: \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:33:54.862239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.862214 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l\" (UID: \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:33:54.923648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.923576 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkj8\" (UniqueName: \"kubernetes.io/projected/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-kube-api-access-hrkj8\") pod \"payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l\" (UID: \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:33:54.929114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:54.929086 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:33:55.323262 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.323162 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l"] Oct 31 00:33:55.349649 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:33:55.345822 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab44c7a3_7f05_4e38_838f_0e8a9ee4c5c8.slice/crio-cf9f61aec91137393c640f5cdfb40d85d8ebb4ad0d930d7aab789f7c06cc88d8 WatchSource:0}: Error finding container cf9f61aec91137393c640f5cdfb40d85d8ebb4ad0d930d7aab789f7c06cc88d8: Status 404 returned error can't find the container with id cf9f61aec91137393c640f5cdfb40d85d8ebb4ad0d930d7aab789f7c06cc88d8 Oct 31 00:33:55.402926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.402891 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t"] Oct 31 00:33:55.408797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.408775 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:33:55.466360 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.466329 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t"] Oct 31 00:33:55.474920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.474894 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b115e833-ff1c-4d0a-be5f-13126f600f21-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t\" (UID: \"b115e833-ff1c-4d0a-be5f-13126f600f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:33:55.475083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.475037 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgzkx\" (UniqueName: \"kubernetes.io/projected/b115e833-ff1c-4d0a-be5f-13126f600f21-kube-api-access-zgzkx\") pod \"payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t\" (UID: \"b115e833-ff1c-4d0a-be5f-13126f600f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:33:55.576945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.576865 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b115e833-ff1c-4d0a-be5f-13126f600f21-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t\" (UID: \"b115e833-ff1c-4d0a-be5f-13126f600f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:33:55.576945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.576940 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgzkx\" (UniqueName: \"kubernetes.io/projected/b115e833-ff1c-4d0a-be5f-13126f600f21-kube-api-access-zgzkx\") pod \"payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t\" (UID: \"b115e833-ff1c-4d0a-be5f-13126f600f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:33:55.577574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.577552 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b115e833-ff1c-4d0a-be5f-13126f600f21-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t\" (UID: \"b115e833-ff1c-4d0a-be5f-13126f600f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:33:55.638373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.638333 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgzkx\" (UniqueName: \"kubernetes.io/projected/b115e833-ff1c-4d0a-be5f-13126f600f21-kube-api-access-zgzkx\") pod \"payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t\" (UID: \"b115e833-ff1c-4d0a-be5f-13126f600f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:33:55.727944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.727912 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:33:55.944088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.944045 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:33:55.944232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:55.944115 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:33:56.113381 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:33:56.113347 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb115e833_ff1c_4d0a_be5f_13126f600f21.slice/crio-9602df7690331dcb543b88b80b4d0645fbc6955dcec9e9300b7248208f9bbaea WatchSource:0}: Error finding container 9602df7690331dcb543b88b80b4d0645fbc6955dcec9e9300b7248208f9bbaea: Status 404 returned error can't find the container with id 9602df7690331dcb543b88b80b4d0645fbc6955dcec9e9300b7248208f9bbaea Oct 31 00:33:56.113905 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.113881 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t"] Oct 31 00:33:56.320974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.320950 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:33:56.347095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.347042 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" event={"ID":"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8","Type":"ContainerStarted","Data":"cf9f61aec91137393c640f5cdfb40d85d8ebb4ad0d930d7aab789f7c06cc88d8"} Oct 31 00:33:56.349253 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.349224 4969 generic.go:356] "Generic (PLEG): container finished" podID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerID="764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898" exitCode=0 Oct 31 00:33:56.349384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.349273 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" event={"ID":"32dde409-51ba-4784-9c04-3f9e8bf45f73","Type":"ContainerDied","Data":"764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898"} Oct 31 00:33:56.349384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.349291 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" Oct 31 00:33:56.349384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.349324 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9" event={"ID":"32dde409-51ba-4784-9c04-3f9e8bf45f73","Type":"ContainerDied","Data":"452b82c9b2d001cc2de210ea0c73bb00bef2f3cbabe353cb190c860db81f5a30"} Oct 31 00:33:56.349384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.349350 4969 scope.go:122] "RemoveContainer" containerID="764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898" Oct 31 00:33:56.350880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.350859 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" event={"ID":"b115e833-ff1c-4d0a-be5f-13126f600f21","Type":"ContainerStarted","Data":"9602df7690331dcb543b88b80b4d0645fbc6955dcec9e9300b7248208f9bbaea"} Oct 31 00:33:56.360238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.360217 4969 scope.go:122] "RemoveContainer" containerID="764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898" Oct 31 00:33:56.360532 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:33:56.360506 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898\": container with ID starting with 764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898 not found: ID does not exist" containerID="764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898" Oct 31 00:33:56.360582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.360539 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898"} err="failed to get container status \"764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898\": rpc error: code = NotFound desc = could not find container \"764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898\": container with ID starting with 764445391a5b8077f7505a6208248574693dc78b77988ced089298175622c898 not found: ID does not exist" Oct 31 00:33:56.386968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.386948 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-certs\") pod \"32dde409-51ba-4784-9c04-3f9e8bf45f73\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " Oct 31 00:33:56.387076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.386989 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-token\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-token\") pod \"32dde409-51ba-4784-9c04-3f9e8bf45f73\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " Oct 31 00:33:56.387076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387025 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32dde409-51ba-4784-9c04-3f9e8bf45f73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32dde409-51ba-4784-9c04-3f9e8bf45f73-istiod-ca-cert\") pod \"32dde409-51ba-4784-9c04-3f9e8bf45f73\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " Oct 31 00:33:56.387193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387094 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-podinfo\") pod \"32dde409-51ba-4784-9c04-3f9e8bf45f73\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " Oct 31 00:33:56.387193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387138 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-envoy\") pod \"32dde409-51ba-4784-9c04-3f9e8bf45f73\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " Oct 31 00:33:56.387296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387233 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-certs" pod "32dde409-51ba-4784-9c04-3f9e8bf45f73" (UID: "32dde409-51ba-4784-9c04-3f9e8bf45f73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.387296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387277 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-data\") pod \"32dde409-51ba-4784-9c04-3f9e8bf45f73\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " Oct 31 00:33:56.387402 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387364 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-kube-api-access-djgq7\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-kube-api-access-djgq7\") pod \"32dde409-51ba-4784-9c04-3f9e8bf45f73\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " Oct 31 00:33:56.387467 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387410 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-socket\") pod \"32dde409-51ba-4784-9c04-3f9e8bf45f73\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " Oct 31 00:33:56.387575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387471 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-credential-socket\") pod \"32dde409-51ba-4784-9c04-3f9e8bf45f73\" (UID: \"32dde409-51ba-4784-9c04-3f9e8bf45f73\") " Oct 31 00:33:56.387575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387478 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32dde409-51ba-4784-9c04-3f9e8bf45f73-istiod-ca-cert" pod "32dde409-51ba-4784-9c04-3f9e8bf45f73" (UID: "32dde409-51ba-4784-9c04-3f9e8bf45f73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:56.387734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387634 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-data" pod "32dde409-51ba-4784-9c04-3f9e8bf45f73" (UID: "32dde409-51ba-4784-9c04-3f9e8bf45f73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.387877 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387837 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-credential-socket" pod "32dde409-51ba-4784-9c04-3f9e8bf45f73" (UID: "32dde409-51ba-4784-9c04-3f9e8bf45f73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.387877 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.387863 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-socket" pod "32dde409-51ba-4784-9c04-3f9e8bf45f73" (UID: "32dde409-51ba-4784-9c04-3f9e8bf45f73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.388363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.388337 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32dde409-51ba-4784-9c04-3f9e8bf45f73-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.388451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.388368 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.388451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.388384 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.388451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.388398 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.388451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.388413 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.389441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.389416 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-podinfo" pod "32dde409-51ba-4784-9c04-3f9e8bf45f73" (UID: "32dde409-51ba-4784-9c04-3f9e8bf45f73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:33:56.389628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.389558 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-envoy" pod "32dde409-51ba-4784-9c04-3f9e8bf45f73" (UID: "32dde409-51ba-4784-9c04-3f9e8bf45f73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:56.389735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.389629 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-token" pod "32dde409-51ba-4784-9c04-3f9e8bf45f73" (UID: "32dde409-51ba-4784-9c04-3f9e8bf45f73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:56.389915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.389898 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-kube-api-access-djgq7" pod "32dde409-51ba-4784-9c04-3f9e8bf45f73" (UID: "32dde409-51ba-4784-9c04-3f9e8bf45f73"). InnerVolumeSpecName "kube-api-access-djgq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:56.489516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.489442 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.489516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.489474 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.489516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.489483 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djgq7\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-kube-api-access-djgq7\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.489516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.489494 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32dde409-51ba-4784-9c04-3f9e8bf45f73-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:33:56.773657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.773553 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9"] Oct 31 00:33:56.778509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:56.778480 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-66bc9"] Oct 31 00:33:57.736109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:33:57.736073 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" path="/var/lib/kubelet/pods/32dde409-51ba-4784-9c04-3f9e8bf45f73/volumes" Oct 31 00:34:00.920535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:00.920502 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2"] Oct 31 00:34:00.921226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:00.921210 4969 cpu_manager.go:414] "containerMap: removing container" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" Oct 31 00:34:00.921272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:00.921227 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" Oct 31 00:34:00.921367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:00.921358 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="32dde409-51ba-4784-9c04-3f9e8bf45f73" containerName="istio-proxy" Oct 31 00:34:00.924939 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:00.924919 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.059297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.059196 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.059297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.059274 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlnq\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-kube-api-access-njlnq\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.059660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.059630 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.059760 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.059710 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.059820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.059756 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.059880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.059858 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.059968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.059915 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391266-6c79-49c5-a25c-e8c48cf07f55-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.060039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.059990 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.060039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.060023 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.066314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.065714 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2"] Oct 31 00:34:01.162005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.161956 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlnq\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-kube-api-access-njlnq\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.162198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.162024 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.162198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.162091 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.162198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.162137 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.162366 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.162236 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.162366 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.162292 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391266-6c79-49c5-a25c-e8c48cf07f55-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.162479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.162369 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.162479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.162402 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.162563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.162500 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.163469 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.163444 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.163739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.163713 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.163968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.163948 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.164682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.164655 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.165141 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.165105 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391266-6c79-49c5-a25c-e8c48cf07f55-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.167043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.167018 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.167221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.167188 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.224492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.224422 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.224878 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.224849 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlnq\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-kube-api-access-njlnq\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.240962 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.240908 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:01.694111 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:34:01.694075 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea391266_6c79_49c5_a25c_e8c48cf07f55.slice/crio-82423d3e4f843d5f0a3eaa6b8cfd8fe98f09c214ddda76b3cca4aa1eb59b5222 WatchSource:0}: Error finding container 82423d3e4f843d5f0a3eaa6b8cfd8fe98f09c214ddda76b3cca4aa1eb59b5222: Status 404 returned error can't find the container with id 82423d3e4f843d5f0a3eaa6b8cfd8fe98f09c214ddda76b3cca4aa1eb59b5222 Oct 31 00:34:01.694504 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.694482 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2"] Oct 31 00:34:01.696809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.696779 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:34:01.696893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.696856 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:34:01.696893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:01.696888 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:34:02.049950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.049914 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z"] Oct 31 00:34:02.054148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.054119 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfead6dd-czl5z"] Oct 31 00:34:02.431655 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.431093 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" event={"ID":"b115e833-ff1c-4d0a-be5f-13126f600f21","Type":"ContainerStarted","Data":"b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b"} Oct 31 00:34:02.431655 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.431165 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:34:02.433351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.433314 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" event={"ID":"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8","Type":"ContainerStarted","Data":"faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21"} Oct 31 00:34:02.433485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.433465 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:34:02.435292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.435268 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" event={"ID":"ea391266-6c79-49c5-a25c-e8c48cf07f55","Type":"ContainerStarted","Data":"de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd"} Oct 31 00:34:02.435410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.435312 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" event={"ID":"ea391266-6c79-49c5-a25c-e8c48cf07f55","Type":"ContainerStarted","Data":"82423d3e4f843d5f0a3eaa6b8cfd8fe98f09c214ddda76b3cca4aa1eb59b5222"} Oct 31 00:34:02.549323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.549263 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" podStartSLOduration=2.142968504 podStartE2EDuration="7.54924522s" totalImagesPullingTime="5.406276716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:02.548622486 +0000 UTC m=+1875.023980906" watchObservedRunningTime="2026-10-31 00:34:02.54924522 +0000 UTC m=+1875.024603645" Oct 31 00:34:02.675631 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.675151 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" podStartSLOduration=2.488038745 podStartE2EDuration="8.675129568s" totalImagesPullingTime="6.187090823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:02.667260253 +0000 UTC m=+1875.142618675" watchObservedRunningTime="2026-10-31 00:34:02.675129568 +0000 UTC m=+1875.150487993" Oct 31 00:34:02.753017 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:02.752915 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:34:02.794628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:02.794112 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" podStartSLOduration=2.794085113 podStartE2EDuration="2.794085113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:02.791934354 +0000 UTC m=+1875.267292777" watchObservedRunningTime="2026-10-31 00:34:02.794085113 +0000 UTC m=+1875.269443574" Oct 31 00:34:03.241294 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:03.241258 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:03.242873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:03.242840 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:34:03.243023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:03.242916 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 00:34:03.736802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:03.736766 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a61db72-53a7-4ab0-8047-4bbcdc86b84c" path="/var/lib/kubelet/pods/7a61db72-53a7-4ab0-8047-4bbcdc86b84c/volumes" Oct 31 00:34:04.241785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:04.241742 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:34:04.242233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:04.241807 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 00:34:05.241529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.241490 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:34:05.241779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.241553 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 00:34:05.525507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.525427 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd"] Oct 31 00:34:05.531014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.530995 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.552638 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.552606 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.552757 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.552741 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f60d2c-d53b-4df2-a835-99f00659294a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.552880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.552863 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7jwh\" (UniqueName: \"kubernetes.io/projected/29f60d2c-d53b-4df2-a835-99f00659294a-kube-api-access-k7jwh\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.552947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.552903 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-metrics-certs\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.610630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.610571 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd"] Oct 31 00:34:05.632362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.632330 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-70babe-serving-cert\"" Oct 31 00:34:05.653752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.653722 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7jwh\" (UniqueName: \"kubernetes.io/projected/29f60d2c-d53b-4df2-a835-99f00659294a-kube-api-access-k7jwh\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.653919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.653758 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-metrics-certs\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.653919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.653870 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.654010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.653951 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f60d2c-d53b-4df2-a835-99f00659294a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.654053 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:05.654002 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-serving-cert: secret "maas-api-e2e-ipp-b-70babe-serving-cert" not found Oct 31 00:34:05.654087 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:05.654070 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls podName:29f60d2c-d53b-4df2-a835-99f00659294a nodeName:}" failed. No retries permitted until 2026-10-31 00:34:06.154054133 +0000 UTC m=+1878.629412552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls") pod "maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" (UID: "29f60d2c-d53b-4df2-a835-99f00659294a") : secret "maas-api-e2e-ipp-b-70babe-serving-cert" not found Oct 31 00:34:05.654613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.654557 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f60d2c-d53b-4df2-a835-99f00659294a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.656478 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.656452 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-metrics-certs\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:05.717476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:05.717444 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7jwh\" (UniqueName: \"kubernetes.io/projected/29f60d2c-d53b-4df2-a835-99f00659294a-kube-api-access-k7jwh\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:06.161444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:06.161406 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:06.164275 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:06.164245 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:06.241891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:06.241854 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:34:06.242092 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:06.241907 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 00:34:06.446970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:06.446882 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:06.857913 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:34:06.857872 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f60d2c_d53b_4df2_a835_99f00659294a.slice/crio-b73cf3df65fcf720467bcf598a9310f0b4a8d963e02de13b38dd132f5beecc7c WatchSource:0}: Error finding container b73cf3df65fcf720467bcf598a9310f0b4a8d963e02de13b38dd132f5beecc7c: Status 404 returned error can't find the container with id b73cf3df65fcf720467bcf598a9310f0b4a8d963e02de13b38dd132f5beecc7c Oct 31 00:34:06.858411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:06.858385 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd"] Oct 31 00:34:07.241912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:07.241869 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 00:34:07.242090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:07.241935 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 00:34:07.503059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:07.502969 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" event={"ID":"29f60d2c-d53b-4df2-a835-99f00659294a","Type":"ContainerStarted","Data":"b73cf3df65fcf720467bcf598a9310f0b4a8d963e02de13b38dd132f5beecc7c"} Oct 31 00:34:08.246962 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:08.246934 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:08.247404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:08.247387 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:08.248140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:08.248119 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:34:08.515244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:08.515151 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" event={"ID":"29f60d2c-d53b-4df2-a835-99f00659294a","Type":"ContainerStarted","Data":"44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026"} Oct 31 00:34:08.742438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:08.742381 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" podStartSLOduration=2.813256076 podStartE2EDuration="3.74236501s" totalImagesPullingTime="929.108934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:08.727322567 +0000 UTC m=+1881.202681026" watchObservedRunningTime="2026-10-31 00:34:08.74236501 +0000 UTC m=+1881.217723433" Oct 31 00:34:09.523824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:09.523794 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:13.446573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:13.446530 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:34:13.446992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:13.446657 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:34:15.540046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:15.540018 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:34:16.041977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.041940 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b"] Oct 31 00:34:16.046471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.046453 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.139585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.139541 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.139790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.139700 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.139790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.139783 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.139938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.139868 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.139938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.139913 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.140029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.139970 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb56f\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-kube-api-access-lb56f\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.140073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.140034 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.140155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.140141 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.140219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.140200 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.140646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.140626 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b"] Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242036 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242127 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242226 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242296 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242372 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242409 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242462 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242523 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb56f\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-kube-api-access-lb56f\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242620 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.242976 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.243355 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.243737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.243709 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.244471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.244254 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.255790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.252178 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.256133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.256105 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.256361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.256339 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.303852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.303787 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.304107 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.304089 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb56f\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-kube-api-access-lb56f\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.360282 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.360246 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:16.800037 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:34:16.800004 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec837b0a_64cd_48c8_9cb0_bebedfd18d26.slice/crio-1015f0c2b4e35152370e097fd7ef81908d7b7db80cb7a3b00c1e8f485fc3a5d7 WatchSource:0}: Error finding container 1015f0c2b4e35152370e097fd7ef81908d7b7db80cb7a3b00c1e8f485fc3a5d7: Status 404 returned error can't find the container with id 1015f0c2b4e35152370e097fd7ef81908d7b7db80cb7a3b00c1e8f485fc3a5d7 Oct 31 00:34:16.800471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.800447 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b"] Oct 31 00:34:16.802784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.802752 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:34:16.802891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.802813 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:34:16.802891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:16.802842 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:34:17.610031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:17.609998 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" event={"ID":"ec837b0a-64cd-48c8-9cb0-bebedfd18d26","Type":"ContainerStarted","Data":"069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f"} Oct 31 00:34:17.610232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:17.610049 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" event={"ID":"ec837b0a-64cd-48c8-9cb0-bebedfd18d26","Type":"ContainerStarted","Data":"1015f0c2b4e35152370e097fd7ef81908d7b7db80cb7a3b00c1e8f485fc3a5d7"} Oct 31 00:34:17.735011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:17.734962 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" podStartSLOduration=2.7349480120000003 podStartE2EDuration="2.734948012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:17.729170205 +0000 UTC m=+1890.204528628" watchObservedRunningTime="2026-10-31 00:34:17.734948012 +0000 UTC m=+1890.210306434" Oct 31 00:34:18.361256 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:18.361218 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:18.362717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:18.362696 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:34:18.362783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:18.362741 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 00:34:19.361053 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:19.361016 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:34:19.361241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:19.361083 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 00:34:20.361327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:20.361289 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:34:20.361730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:20.361343 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 00:34:21.361711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:21.361674 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:34:21.362138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:21.361729 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 00:34:22.361487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:22.361455 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:34:22.361682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:22.361506 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 00:34:22.440898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:22.440862 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz"] Oct 31 00:34:22.470641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:22.470613 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-835921-xhdzz"] Oct 31 00:34:23.398908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.398870 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7504-5eab08"] Oct 31 00:34:23.404331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.404307 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:23.404496 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.404409 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:23.404864 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.404832 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:23.405734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.405715 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:34:23.459365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.459334 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-5eab08"] Oct 31 00:34:23.468003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.467971 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t922\" (UniqueName: \"kubernetes.io/projected/e037e7f2-6470-4fa0-a1cc-1c741b8d33c7-kube-api-access-7t922\") pod \"test-curl-7504-5eab08\" (UID: \"e037e7f2-6470-4fa0-a1cc-1c741b8d33c7\") " pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:23.506040 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.506010 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9d9vc\"" Oct 31 00:34:23.569896 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.569861 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t922\" (UniqueName: \"kubernetes.io/projected/e037e7f2-6470-4fa0-a1cc-1c741b8d33c7-kube-api-access-7t922\") pod \"test-curl-7504-5eab08\" (UID: \"e037e7f2-6470-4fa0-a1cc-1c741b8d33c7\") " pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:23.631079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.631042 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t922\" (UniqueName: \"kubernetes.io/projected/e037e7f2-6470-4fa0-a1cc-1c741b8d33c7-kube-api-access-7t922\") pod \"test-curl-7504-5eab08\" (UID: \"e037e7f2-6470-4fa0-a1cc-1c741b8d33c7\") " pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:23.724954 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.724861 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:23.743743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:23.743710 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="808dac3b-1afd-496e-923a-80be5c1605be" path="/var/lib/kubelet/pods/808dac3b-1afd-496e-923a-80be5c1605be/volumes" Oct 31 00:34:24.122130 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:34:24.122101 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode037e7f2_6470_4fa0_a1cc_1c741b8d33c7.slice/crio-a2feeb65743899542d7cb0f9ce4032136a859d486e575b08d00270d6f9847ff0 WatchSource:0}: Error finding container a2feeb65743899542d7cb0f9ce4032136a859d486e575b08d00270d6f9847ff0: Status 404 returned error can't find the container with id a2feeb65743899542d7cb0f9ce4032136a859d486e575b08d00270d6f9847ff0 Oct 31 00:34:24.122284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:24.122192 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-5eab08"] Oct 31 00:34:24.671853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:24.671819 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-5eab08" event={"ID":"e037e7f2-6470-4fa0-a1cc-1c741b8d33c7","Type":"ContainerStarted","Data":"bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863"} Oct 31 00:34:24.672279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:24.671861 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-5eab08" event={"ID":"e037e7f2-6470-4fa0-a1cc-1c741b8d33c7","Type":"ContainerStarted","Data":"a2feeb65743899542d7cb0f9ce4032136a859d486e575b08d00270d6f9847ff0"} Oct 31 00:34:24.788516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:24.788463 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7504-5eab08" podStartSLOduration=1.788445675 podStartE2EDuration="1.788445675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:24.78425464 +0000 UTC m=+1897.259613066" watchObservedRunningTime="2026-10-31 00:34:24.788445675 +0000 UTC m=+1897.263804100" Oct 31 00:34:26.324695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:26.324572 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7504-5eab08"] Oct 31 00:34:26.331340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:26.331310 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7504-5eab08"] Oct 31 00:34:26.689532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:26.689489 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7504-5eab08" podUID="e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" containerName="test-curl-7504-5eab08" containerID="cri-o://bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863" gracePeriod=2 Oct 31 00:34:26.740677 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:26.740639 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7504-5eab08\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:27.794045 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:27.794000 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7504-5eab08\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:28.965077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:28.965014 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:29.047249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.047211 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e037e7f2-6470-4fa0-a1cc-1c741b8d33c7-kube-api-access-7t922\" (UniqueName: \"kubernetes.io/projected/e037e7f2-6470-4fa0-a1cc-1c741b8d33c7-kube-api-access-7t922\") pod \"e037e7f2-6470-4fa0-a1cc-1c741b8d33c7\" (UID: \"e037e7f2-6470-4fa0-a1cc-1c741b8d33c7\") " Oct 31 00:34:29.050648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.050247 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e037e7f2-6470-4fa0-a1cc-1c741b8d33c7-kube-api-access-7t922" pod "e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" (UID: "e037e7f2-6470-4fa0-a1cc-1c741b8d33c7"). InnerVolumeSpecName "kube-api-access-7t922". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:29.123644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.123613 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn"] Oct 31 00:34:29.123827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.123805 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" containerID="cri-o://3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b" gracePeriod=30 Oct 31 00:34:29.149035 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.149002 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t922\" (UniqueName: \"kubernetes.io/projected/e037e7f2-6470-4fa0-a1cc-1c741b8d33c7-kube-api-access-7t922\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:29.715876 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.715837 4969 generic.go:356] "Generic (PLEG): container finished" podID="e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" containerID="bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863" exitCode=137 Oct 31 00:34:29.715876 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.715882 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:29.716117 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.715930 4969 scope.go:122] "RemoveContainer" containerID="bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863" Oct 31 00:34:29.728100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.728082 4969 scope.go:122] "RemoveContainer" containerID="bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863" Oct 31 00:34:29.728425 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:29.728405 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863\": container with ID starting with bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863 not found: ID does not exist" containerID="bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863" Oct 31 00:34:29.728487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.728434 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863"} err="failed to get container status \"bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863\": rpc error: code = NotFound desc = could not find container \"bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863\": container with ID starting with bd1ab5a02b51c8fd58ee5a93afc46d4173848bed4ffcfee3bd59e657d2ac1863 not found: ID does not exist" Oct 31 00:34:29.736469 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:29.736446 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" path="/var/lib/kubelet/pods/e037e7f2-6470-4fa0-a1cc-1c741b8d33c7/volumes" Oct 31 00:34:29.775915 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:29.775880 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7504-5eab08\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" pod="opendatahub/test-curl-7504-5eab08" Oct 31 00:34:30.083423 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:30.083350 4969 patch_prober.go:29] interesting pod/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:34:30.083798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:30.083430 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:32.736560 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:32.736510 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:34:34.415322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.415300 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:34:34.542370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542331 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-certs\") pod \"05e6a1ab-f335-42fc-a645-b18f8d554ead\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " Oct 31 00:34:34.542529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542397 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-podinfo\") pod \"05e6a1ab-f335-42fc-a645-b18f8d554ead\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " Oct 31 00:34:34.542576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542531 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-socket\") pod \"05e6a1ab-f335-42fc-a645-b18f8d554ead\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " Oct 31 00:34:34.542576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542556 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-token\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-token\") pod \"05e6a1ab-f335-42fc-a645-b18f8d554ead\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " Oct 31 00:34:34.542691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542583 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-kube-api-access-fcj2p\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-kube-api-access-fcj2p\") pod \"05e6a1ab-f335-42fc-a645-b18f8d554ead\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " Oct 31 00:34:34.542747 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542708 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-certs" pod "05e6a1ab-f335-42fc-a645-b18f8d554ead" (UID: "05e6a1ab-f335-42fc-a645-b18f8d554ead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:34.542836 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542817 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-credential-socket\") pod \"05e6a1ab-f335-42fc-a645-b18f8d554ead\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " Oct 31 00:34:34.542901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542843 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-socket" pod "05e6a1ab-f335-42fc-a645-b18f8d554ead" (UID: "05e6a1ab-f335-42fc-a645-b18f8d554ead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:34.542960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542904 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-envoy\") pod \"05e6a1ab-f335-42fc-a645-b18f8d554ead\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " Oct 31 00:34:34.542960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.542949 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-data\") pod \"05e6a1ab-f335-42fc-a645-b18f8d554ead\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " Oct 31 00:34:34.543072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.543039 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e6a1ab-f335-42fc-a645-b18f8d554ead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6a1ab-f335-42fc-a645-b18f8d554ead-istiod-ca-cert\") pod \"05e6a1ab-f335-42fc-a645-b18f8d554ead\" (UID: \"05e6a1ab-f335-42fc-a645-b18f8d554ead\") " Oct 31 00:34:34.543129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.543114 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-credential-socket" pod "05e6a1ab-f335-42fc-a645-b18f8d554ead" (UID: "05e6a1ab-f335-42fc-a645-b18f8d554ead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:34.543564 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.543253 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-data" pod "05e6a1ab-f335-42fc-a645-b18f8d554ead" (UID: "05e6a1ab-f335-42fc-a645-b18f8d554ead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:34.543893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.543701 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:34.543893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.543729 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:34.543893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.543745 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:34.543893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.543761 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:34.543893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.543852 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e6a1ab-f335-42fc-a645-b18f8d554ead-istiod-ca-cert" pod "05e6a1ab-f335-42fc-a645-b18f8d554ead" (UID: "05e6a1ab-f335-42fc-a645-b18f8d554ead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:34.545212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.545188 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-token" pod "05e6a1ab-f335-42fc-a645-b18f8d554ead" (UID: "05e6a1ab-f335-42fc-a645-b18f8d554ead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:34.545383 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.545364 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-kube-api-access-fcj2p" pod "05e6a1ab-f335-42fc-a645-b18f8d554ead" (UID: "05e6a1ab-f335-42fc-a645-b18f8d554ead"). InnerVolumeSpecName "kube-api-access-fcj2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:34.545508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.545487 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-envoy" pod "05e6a1ab-f335-42fc-a645-b18f8d554ead" (UID: "05e6a1ab-f335-42fc-a645-b18f8d554ead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:34.545584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.545490 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-podinfo" pod "05e6a1ab-f335-42fc-a645-b18f8d554ead" (UID: "05e6a1ab-f335-42fc-a645-b18f8d554ead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:34:34.645066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.644991 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:34.645066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.645019 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:34.645066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.645030 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcj2p\" (UniqueName: \"kubernetes.io/projected/05e6a1ab-f335-42fc-a645-b18f8d554ead-kube-api-access-fcj2p\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:34.645066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.645039 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e6a1ab-f335-42fc-a645-b18f8d554ead-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:34.645066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.645047 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e6a1ab-f335-42fc-a645-b18f8d554ead-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:34.765860 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.765828 4969 generic.go:356] "Generic (PLEG): container finished" podID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerID="3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b" exitCode=0 Oct 31 00:34:34.766042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.765868 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" event={"ID":"05e6a1ab-f335-42fc-a645-b18f8d554ead","Type":"ContainerDied","Data":"3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b"} Oct 31 00:34:34.766042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.765896 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" Oct 31 00:34:34.766042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.765921 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn" event={"ID":"05e6a1ab-f335-42fc-a645-b18f8d554ead","Type":"ContainerDied","Data":"5fbacdee4709d5df5b5776144f220d4b11cc44f2d2ed20bb2f0a9216f4740247"} Oct 31 00:34:34.766042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.765942 4969 scope.go:122] "RemoveContainer" containerID="3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b" Oct 31 00:34:34.780122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.780101 4969 scope.go:122] "RemoveContainer" containerID="1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82" Oct 31 00:34:34.802734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.802703 4969 scope.go:122] "RemoveContainer" containerID="3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b" Oct 31 00:34:34.803320 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:34.803261 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b\": container with ID starting with 3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b not found: ID does not exist" containerID="3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b" Oct 31 00:34:34.803451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.803312 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b"} err="failed to get container status \"3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b\": rpc error: code = NotFound desc = could not find container \"3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b\": container with ID starting with 3f7d6709651f7d29439eca66f062ac4ad0f282ee9246c20593f0ae23e37c7e2b not found: ID does not exist" Oct 31 00:34:34.803451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.803342 4969 scope.go:122] "RemoveContainer" containerID="1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82" Oct 31 00:34:34.804343 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:34.804180 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82\": container with ID starting with 1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82 not found: ID does not exist" containerID="1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82" Oct 31 00:34:34.804343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.804212 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82"} err="failed to get container status \"1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82\": rpc error: code = NotFound desc = could not find container \"1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82\": container with ID starting with 1e2bc9a0006d48de9e46bad9340d676056a8decb92e25f081e52723d087a4d82 not found: ID does not exist" Oct 31 00:34:34.946224 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.946132 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn"] Oct 31 00:34:34.950463 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:34.950437 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-835921-openshift-default-58b55c7f6f-l7tbn"] Oct 31 00:34:35.735994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:35.735957 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" path="/var/lib/kubelet/pods/05e6a1ab-f335-42fc-a645-b18f8d554ead/volumes" Oct 31 00:34:38.575408 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:38.575376 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b"] Oct 31 00:34:41.211430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:41.211389 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs"] Oct 31 00:34:43.576199 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.576153 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" podUID="d6934960-0fb3-42d3-9506-586edae56ad0" containerName="payload-pre-processing" containerID="cri-o://e3f0aff8db4e601eb45f0ea4dd75509d9da7431af3848171dcc28834be9f7b09" gracePeriod=25 Oct 31 00:34:43.872660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.872623 4969 generic.go:356] "Generic (PLEG): container finished" podID="d6934960-0fb3-42d3-9506-586edae56ad0" containerID="e3f0aff8db4e601eb45f0ea4dd75509d9da7431af3848171dcc28834be9f7b09" exitCode=0 Oct 31 00:34:43.873633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.872646 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" event={"ID":"d6934960-0fb3-42d3-9506-586edae56ad0","Type":"ContainerDied","Data":"e3f0aff8db4e601eb45f0ea4dd75509d9da7431af3848171dcc28834be9f7b09"} Oct 31 00:34:43.873900 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.873842 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" event={"ID":"d6934960-0fb3-42d3-9506-586edae56ad0","Type":"ContainerDied","Data":"e2ea06166539d9e13c3366e2f1c187a41d0b43e8c435fd7d2254fe0f8ee3b3f7"} Oct 31 00:34:43.873900 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.873880 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ea06166539d9e13c3366e2f1c187a41d0b43e8c435fd7d2254fe0f8ee3b3f7" Oct 31 00:34:43.881156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.881123 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:34:43.925444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.925411 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6934960-0fb3-42d3-9506-586edae56ad0-kube-api-access-mwxgq\" (UniqueName: \"kubernetes.io/projected/d6934960-0fb3-42d3-9506-586edae56ad0-kube-api-access-mwxgq\") pod \"d6934960-0fb3-42d3-9506-586edae56ad0\" (UID: \"d6934960-0fb3-42d3-9506-586edae56ad0\") " Oct 31 00:34:43.925652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.925468 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6934960-0fb3-42d3-9506-586edae56ad0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6934960-0fb3-42d3-9506-586edae56ad0-plugins-config-volume\") pod \"d6934960-0fb3-42d3-9506-586edae56ad0\" (UID: \"d6934960-0fb3-42d3-9506-586edae56ad0\") " Oct 31 00:34:43.925973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.925945 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6934960-0fb3-42d3-9506-586edae56ad0-plugins-config-volume" pod "d6934960-0fb3-42d3-9506-586edae56ad0" (UID: "d6934960-0fb3-42d3-9506-586edae56ad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:43.926163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.926144 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6934960-0fb3-42d3-9506-586edae56ad0-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:43.928110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:43.928077 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6934960-0fb3-42d3-9506-586edae56ad0-kube-api-access-mwxgq" pod "d6934960-0fb3-42d3-9506-586edae56ad0" (UID: "d6934960-0fb3-42d3-9506-586edae56ad0"). InnerVolumeSpecName "kube-api-access-mwxgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:44.028133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:44.028085 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwxgq\" (UniqueName: \"kubernetes.io/projected/d6934960-0fb3-42d3-9506-586edae56ad0-kube-api-access-mwxgq\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:44.883174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:44.883144 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b" Oct 31 00:34:45.049486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:45.049455 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b"] Oct 31 00:34:45.098819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:45.098779 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-dgh8b"] Oct 31 00:34:45.574738 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:45.574692 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:34:45.574948 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:45.574753 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:34:45.737144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:45.737103 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6934960-0fb3-42d3-9506-586edae56ad0" path="/var/lib/kubelet/pods/d6934960-0fb3-42d3-9506-586edae56ad0/volumes" Oct 31 00:34:46.212470 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.212382 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" podUID="5752d604-f290-4a9d-b73a-4d8852d02864" containerName="payload-pre-processing" containerID="cri-o://566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998" gracePeriod=25 Oct 31 00:34:46.548574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.548549 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:34:46.666011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.665978 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5752d604-f290-4a9d-b73a-4d8852d02864-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5752d604-f290-4a9d-b73a-4d8852d02864-plugins-config-volume\") pod \"5752d604-f290-4a9d-b73a-4d8852d02864\" (UID: \"5752d604-f290-4a9d-b73a-4d8852d02864\") " Oct 31 00:34:46.666184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.666111 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5752d604-f290-4a9d-b73a-4d8852d02864-kube-api-access-q8jj9\" (UniqueName: \"kubernetes.io/projected/5752d604-f290-4a9d-b73a-4d8852d02864-kube-api-access-q8jj9\") pod \"5752d604-f290-4a9d-b73a-4d8852d02864\" (UID: \"5752d604-f290-4a9d-b73a-4d8852d02864\") " Oct 31 00:34:46.666961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.666931 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5752d604-f290-4a9d-b73a-4d8852d02864-plugins-config-volume" pod "5752d604-f290-4a9d-b73a-4d8852d02864" (UID: "5752d604-f290-4a9d-b73a-4d8852d02864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:46.669303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.669275 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5752d604-f290-4a9d-b73a-4d8852d02864-kube-api-access-q8jj9" pod "5752d604-f290-4a9d-b73a-4d8852d02864" (UID: "5752d604-f290-4a9d-b73a-4d8852d02864"). InnerVolumeSpecName "kube-api-access-q8jj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:46.767873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.767790 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8jj9\" (UniqueName: \"kubernetes.io/projected/5752d604-f290-4a9d-b73a-4d8852d02864-kube-api-access-q8jj9\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:46.767873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.767819 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5752d604-f290-4a9d-b73a-4d8852d02864-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:34:46.905677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.905631 4969 generic.go:356] "Generic (PLEG): container finished" podID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerID="81464f271916c299164d58c66e6365530f0aeae425794f54e7375d823e51096c" exitCode=0 Oct 31 00:34:46.905859 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.905700 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" event={"ID":"13f2becb-a214-43c1-8d2c-519a00bfd076","Type":"ContainerDied","Data":"81464f271916c299164d58c66e6365530f0aeae425794f54e7375d823e51096c"} Oct 31 00:34:46.906205 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.906186 4969 scope.go:122] "RemoveContainer" containerID="81464f271916c299164d58c66e6365530f0aeae425794f54e7375d823e51096c" Oct 31 00:34:46.907368 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.907343 4969 generic.go:356] "Generic (PLEG): container finished" podID="5752d604-f290-4a9d-b73a-4d8852d02864" containerID="566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998" exitCode=0 Oct 31 00:34:46.907464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.907411 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" event={"ID":"5752d604-f290-4a9d-b73a-4d8852d02864","Type":"ContainerDied","Data":"566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998"} Oct 31 00:34:46.907464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.907456 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" event={"ID":"5752d604-f290-4a9d-b73a-4d8852d02864","Type":"ContainerDied","Data":"8b253b8e1e89d4b276a0429d4d615dad55cee771b0a987922458c2393b91ddb1"} Oct 31 00:34:46.907579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.907462 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs" Oct 31 00:34:46.907579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.907484 4969 scope.go:122] "RemoveContainer" containerID="566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998" Oct 31 00:34:46.907901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.907872 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:34:46.907992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.907960 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:34:46.908053 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.908002 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:34:46.925442 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.925415 4969 scope.go:122] "RemoveContainer" containerID="566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998" Oct 31 00:34:46.925805 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:34:46.925782 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998\": container with ID starting with 566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998 not found: ID does not exist" containerID="566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998" Oct 31 00:34:46.925880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:46.925817 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998"} err="failed to get container status \"566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998\": rpc error: code = NotFound desc = could not find container \"566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998\": container with ID starting with 566210d57a3cabf492eff96a808f98cf8882eb4186b19b3446cb12247e27b998 not found: ID does not exist" Oct 31 00:34:47.152036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:47.151814 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs"] Oct 31 00:34:47.156721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:47.156685 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a59a7-8b96b4675-hpfjs"] Oct 31 00:34:47.737323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:47.737293 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5752d604-f290-4a9d-b73a-4d8852d02864" path="/var/lib/kubelet/pods/5752d604-f290-4a9d-b73a-4d8852d02864/volumes" Oct 31 00:34:47.917410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:47.917376 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" event={"ID":"13f2becb-a214-43c1-8d2c-519a00bfd076","Type":"ContainerStarted","Data":"fe3022a942ef3bbe1786e33a5736bf3a69431635547197ca86e12a6277ecbad7"} Oct 31 00:34:47.928789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:47.928753 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:34:47.929015 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:47.928932 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:34:47.929015 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:47.928989 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 00:34:48.861361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:48.861331 4969 scope.go:122] "RemoveContainer" containerID="da5fe9c2fde4acf0532144cf6ed979ae3b980667ca538eca8f4e633f579cde51" Oct 31 00:34:48.928466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:48.928434 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:34:48.928534 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:48.928506 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 00:34:49.928509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:49.928474 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:34:49.928960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:49.928536 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 00:34:50.049666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:50.049631 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss"] Oct 31 00:34:50.106079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:50.106047 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d19ef0b3-ptjss"] Oct 31 00:34:50.928569 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:50.928534 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:34:50.929003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:50.928630 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 00:34:51.736213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:51.736179 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e" path="/var/lib/kubelet/pods/91d303ad-1dc5-4fc0-999a-0dcfc8b3bb2e/volumes" Oct 31 00:34:51.929094 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:51.929055 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:34:51.929573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:51.929119 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 00:34:52.928566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:52.928530 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 31 00:34:52.928768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:52.928602 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 31 00:34:53.933140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:53.933110 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:34:53.933649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:53.933628 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:34:53.934453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:34:53.934429 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:35:00.531871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:00.529505 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g"] Oct 31 00:35:00.635343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:00.635307 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f"] Oct 31 00:35:02.733198 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:02.733161 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:35:05.530128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.530069 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" podUID="d75a9303-d87f-4ba8-81e1-2235dcf7685b" containerName="payload-processing" containerID="cri-o://680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952" gracePeriod=25 Oct 31 00:35:05.635661 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.635572 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" podUID="acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec" containerName="payload-pre-processing" containerID="cri-o://7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe" gracePeriod=25 Oct 31 00:35:05.842001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.841970 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:35:05.929857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.929832 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:35:05.963474 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.963446 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75a9303-d87f-4ba8-81e1-2235dcf7685b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75a9303-d87f-4ba8-81e1-2235dcf7685b-plugins-config-volume\") pod \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\" (UID: \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\") " Oct 31 00:35:05.963667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.963495 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-kube-api-access-75dmd\" (UniqueName: \"kubernetes.io/projected/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-kube-api-access-75dmd\") pod \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\" (UID: \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\") " Oct 31 00:35:05.963667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.963615 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-plugins-config-volume\") pod \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\" (UID: \"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec\") " Oct 31 00:35:05.963812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.963772 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75a9303-d87f-4ba8-81e1-2235dcf7685b-kube-api-access-cqvgf\" (UniqueName: \"kubernetes.io/projected/d75a9303-d87f-4ba8-81e1-2235dcf7685b-kube-api-access-cqvgf\") pod \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\" (UID: \"d75a9303-d87f-4ba8-81e1-2235dcf7685b\") " Oct 31 00:35:05.963986 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.963963 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75a9303-d87f-4ba8-81e1-2235dcf7685b-plugins-config-volume" pod "d75a9303-d87f-4ba8-81e1-2235dcf7685b" (UID: "d75a9303-d87f-4ba8-81e1-2235dcf7685b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:05.964100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.964058 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-plugins-config-volume" pod "acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec" (UID: "acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:05.964475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.964418 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75a9303-d87f-4ba8-81e1-2235dcf7685b-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:05.964475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.964448 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:05.966133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.966108 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-kube-api-access-75dmd" pod "acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec" (UID: "acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec"). InnerVolumeSpecName "kube-api-access-75dmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:05.966447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:05.966426 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75a9303-d87f-4ba8-81e1-2235dcf7685b-kube-api-access-cqvgf" pod "d75a9303-d87f-4ba8-81e1-2235dcf7685b" (UID: "d75a9303-d87f-4ba8-81e1-2235dcf7685b"). InnerVolumeSpecName "kube-api-access-cqvgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:06.065545 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.065454 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqvgf\" (UniqueName: \"kubernetes.io/projected/d75a9303-d87f-4ba8-81e1-2235dcf7685b-kube-api-access-cqvgf\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:06.065545 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.065488 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75dmd\" (UniqueName: \"kubernetes.io/projected/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec-kube-api-access-75dmd\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:06.107675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.107633 4969 generic.go:356] "Generic (PLEG): container finished" podID="d75a9303-d87f-4ba8-81e1-2235dcf7685b" containerID="680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952" exitCode=0 Oct 31 00:35:06.107862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.107696 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" Oct 31 00:35:06.107862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.107723 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" event={"ID":"d75a9303-d87f-4ba8-81e1-2235dcf7685b","Type":"ContainerDied","Data":"680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952"} Oct 31 00:35:06.107862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.107789 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g" event={"ID":"d75a9303-d87f-4ba8-81e1-2235dcf7685b","Type":"ContainerDied","Data":"aa8aec9c2fe73b077cb9ca785432ae89a10a8fdda9e426adb233ce5299b6532f"} Oct 31 00:35:06.107862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.107818 4969 scope.go:122] "RemoveContainer" containerID="680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952" Oct 31 00:35:06.109911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.109881 4969 generic.go:356] "Generic (PLEG): container finished" podID="acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec" containerID="7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe" exitCode=0 Oct 31 00:35:06.110116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.109926 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" Oct 31 00:35:06.110116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.109991 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" event={"ID":"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec","Type":"ContainerDied","Data":"7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe"} Oct 31 00:35:06.110116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.110053 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f" event={"ID":"acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec","Type":"ContainerDied","Data":"93777f5a2fbf49f10b97b9b713a10c8776582115118dbf2fc15b10daf10b92a6"} Oct 31 00:35:06.126649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.126627 4969 scope.go:122] "RemoveContainer" containerID="680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952" Oct 31 00:35:06.126976 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:06.126951 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952\": container with ID starting with 680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952 not found: ID does not exist" containerID="680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952" Oct 31 00:35:06.127078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.126985 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952"} err="failed to get container status \"680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952\": rpc error: code = NotFound desc = could not find container \"680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952\": container with ID starting with 680d1fdf965fa70c0aaa16cef7a562ee71e87f9d32f7caba49cf96f50e91a952 not found: ID does not exist" Oct 31 00:35:06.127078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.127009 4969 scope.go:122] "RemoveContainer" containerID="7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe" Oct 31 00:35:06.136581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.136565 4969 scope.go:122] "RemoveContainer" containerID="7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe" Oct 31 00:35:06.136867 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:06.136848 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe\": container with ID starting with 7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe not found: ID does not exist" containerID="7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe" Oct 31 00:35:06.136931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.136875 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe"} err="failed to get container status \"7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe\": rpc error: code = NotFound desc = could not find container \"7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe\": container with ID starting with 7d50448092de0f35472c31c62411e3ad0134432313eb36d7fb53e9f2a4fda0fe not found: ID does not exist" Oct 31 00:35:06.275202 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.275174 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g"] Oct 31 00:35:06.283815 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.283786 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e7872-67cf679bc9-fq47g"] Oct 31 00:35:06.394772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.394734 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f"] Oct 31 00:35:06.399866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:06.399842 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e7872-65f9df4c8c-c5q8f"] Oct 31 00:35:07.735922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:07.735890 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec" path="/var/lib/kubelet/pods/acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec/volumes" Oct 31 00:35:07.736324 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:07.736310 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75a9303-d87f-4ba8-81e1-2235dcf7685b" path="/var/lib/kubelet/pods/d75a9303-d87f-4ba8-81e1-2235dcf7685b/volumes" Oct 31 00:35:14.328616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.328181 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb"] Oct 31 00:35:14.328616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.328418 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" podUID="1118a188-069d-4f54-b6a9-fc9cd626ef21" containerName="maas-api" containerID="cri-o://66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513" gracePeriod=30 Oct 31 00:35:14.752674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.752647 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:35:14.931680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.931649 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1118a188-069d-4f54-b6a9-fc9cd626ef21-kube-api-access-v4vpv\" (UniqueName: \"kubernetes.io/projected/1118a188-069d-4f54-b6a9-fc9cd626ef21-kube-api-access-v4vpv\") pod \"1118a188-069d-4f54-b6a9-fc9cd626ef21\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " Oct 31 00:35:14.931889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.931792 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1118a188-069d-4f54-b6a9-fc9cd626ef21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1118a188-069d-4f54-b6a9-fc9cd626ef21-openshift-service-ca\") pod \"1118a188-069d-4f54-b6a9-fc9cd626ef21\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " Oct 31 00:35:14.932020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.932001 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls\") pod \"1118a188-069d-4f54-b6a9-fc9cd626ef21\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " Oct 31 00:35:14.932111 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.932090 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-metrics-certs\") pod \"1118a188-069d-4f54-b6a9-fc9cd626ef21\" (UID: \"1118a188-069d-4f54-b6a9-fc9cd626ef21\") " Oct 31 00:35:14.932235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.932110 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1118a188-069d-4f54-b6a9-fc9cd626ef21-openshift-service-ca" pod "1118a188-069d-4f54-b6a9-fc9cd626ef21" (UID: "1118a188-069d-4f54-b6a9-fc9cd626ef21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:14.932754 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.932672 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1118a188-069d-4f54-b6a9-fc9cd626ef21-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:14.934087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.934062 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1118a188-069d-4f54-b6a9-fc9cd626ef21-kube-api-access-v4vpv" pod "1118a188-069d-4f54-b6a9-fc9cd626ef21" (UID: "1118a188-069d-4f54-b6a9-fc9cd626ef21"). InnerVolumeSpecName "kube-api-access-v4vpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:14.934260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.934241 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-metrics-certs" pod "1118a188-069d-4f54-b6a9-fc9cd626ef21" (UID: "1118a188-069d-4f54-b6a9-fc9cd626ef21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:14.934309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:14.934261 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls" pod "1118a188-069d-4f54-b6a9-fc9cd626ef21" (UID: "1118a188-069d-4f54-b6a9-fc9cd626ef21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:15.033972 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.033886 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:15.033972 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.033922 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vpv\" (UniqueName: \"kubernetes.io/projected/1118a188-069d-4f54-b6a9-fc9cd626ef21-kube-api-access-v4vpv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:15.033972 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.033936 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1118a188-069d-4f54-b6a9-fc9cd626ef21-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:15.185511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.185476 4969 generic.go:356] "Generic (PLEG): container finished" podID="1118a188-069d-4f54-b6a9-fc9cd626ef21" containerID="66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513" exitCode=0 Oct 31 00:35:15.185702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.185530 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" Oct 31 00:35:15.185702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.185564 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" event={"ID":"1118a188-069d-4f54-b6a9-fc9cd626ef21","Type":"ContainerDied","Data":"66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513"} Oct 31 00:35:15.185702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.185650 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb" event={"ID":"1118a188-069d-4f54-b6a9-fc9cd626ef21","Type":"ContainerDied","Data":"bac94ddce7184985b160f16878913ad6e47eaae520f464ff0886c6257e16974b"} Oct 31 00:35:15.185702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.185673 4969 scope.go:122] "RemoveContainer" containerID="66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513" Oct 31 00:35:15.197428 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.197410 4969 scope.go:122] "RemoveContainer" containerID="66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513" Oct 31 00:35:15.197732 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:15.197710 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513\": container with ID starting with 66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513 not found: ID does not exist" containerID="66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513" Oct 31 00:35:15.197808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.197744 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513"} err="failed to get container status \"66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513\": rpc error: code = NotFound desc = could not find container \"66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513\": container with ID starting with 66a42f7d41d2f29c8c41cdcefa64ae6b90494b8927800933ec2169a76c620513 not found: ID does not exist" Oct 31 00:35:15.387070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.387035 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb"] Oct 31 00:35:15.391107 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.391082 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-qgcnb"] Oct 31 00:35:15.735793 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:15.735711 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1118a188-069d-4f54-b6a9-fc9cd626ef21" path="/var/lib/kubelet/pods/1118a188-069d-4f54-b6a9-fc9cd626ef21/volumes" Oct 31 00:35:19.948622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:19.948571 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds"] Oct 31 00:35:20.059902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:20.059867 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv"] Oct 31 00:35:24.949769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:24.949720 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" podUID="74a85cff-6b8c-41de-b5dc-31b0e048dd26" containerName="payload-processing" containerID="cri-o://1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079" gracePeriod=25 Oct 31 00:35:25.060660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.060619 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" podUID="3dab119b-ecff-4560-a864-880e3b8c87da" containerName="payload-pre-processing" containerID="cri-o://2d7da94deb0fd2e39da7fc06227652d2acbea750f21cca65af5c9b1c85bf4c4e" gracePeriod=25 Oct 31 00:35:25.265919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.265887 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:35:25.325927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.325895 4969 generic.go:356] "Generic (PLEG): container finished" podID="74a85cff-6b8c-41de-b5dc-31b0e048dd26" containerID="1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079" exitCode=0 Oct 31 00:35:25.326084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.325938 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" Oct 31 00:35:25.326084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.325972 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" event={"ID":"74a85cff-6b8c-41de-b5dc-31b0e048dd26","Type":"ContainerDied","Data":"1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079"} Oct 31 00:35:25.326084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.326030 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds" event={"ID":"74a85cff-6b8c-41de-b5dc-31b0e048dd26","Type":"ContainerDied","Data":"3bb3242b076ddb69e402db648d2a3faa5d1de21871bbfe49f70ccb3bb92bd242"} Oct 31 00:35:25.326084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.326063 4969 scope.go:122] "RemoveContainer" containerID="1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079" Oct 31 00:35:25.327761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.327735 4969 generic.go:356] "Generic (PLEG): container finished" podID="3dab119b-ecff-4560-a864-880e3b8c87da" containerID="2d7da94deb0fd2e39da7fc06227652d2acbea750f21cca65af5c9b1c85bf4c4e" exitCode=0 Oct 31 00:35:25.327888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.327797 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" event={"ID":"3dab119b-ecff-4560-a864-880e3b8c87da","Type":"ContainerDied","Data":"2d7da94deb0fd2e39da7fc06227652d2acbea750f21cca65af5c9b1c85bf4c4e"} Oct 31 00:35:25.327888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.327837 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" event={"ID":"3dab119b-ecff-4560-a864-880e3b8c87da","Type":"ContainerDied","Data":"112737d7c4f24a9b003293c8f63fdac455142ea0a902ac1980b65a5b93741508"} Oct 31 00:35:25.327888 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.327856 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112737d7c4f24a9b003293c8f63fdac455142ea0a902ac1980b65a5b93741508" Oct 31 00:35:25.341401 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.341384 4969 scope.go:122] "RemoveContainer" containerID="1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079" Oct 31 00:35:25.341765 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:25.341737 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079\": container with ID starting with 1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079 not found: ID does not exist" containerID="1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079" Oct 31 00:35:25.341902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.341772 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079"} err="failed to get container status \"1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079\": rpc error: code = NotFound desc = could not find container \"1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079\": container with ID starting with 1d8d7abfc94792f5fce83e9b7a65b88808ca20562616d7ec8f41dcaa4eeed079 not found: ID does not exist" Oct 31 00:35:25.345393 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.345373 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:35:25.362530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.362510 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a85cff-6b8c-41de-b5dc-31b0e048dd26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a85cff-6b8c-41de-b5dc-31b0e048dd26-plugins-config-volume\") pod \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\" (UID: \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\") " Oct 31 00:35:25.362641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.362625 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a85cff-6b8c-41de-b5dc-31b0e048dd26-kube-api-access-9dqjw\" (UniqueName: \"kubernetes.io/projected/74a85cff-6b8c-41de-b5dc-31b0e048dd26-kube-api-access-9dqjw\") pod \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\" (UID: \"74a85cff-6b8c-41de-b5dc-31b0e048dd26\") " Oct 31 00:35:25.362923 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.362901 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a85cff-6b8c-41de-b5dc-31b0e048dd26-plugins-config-volume" pod "74a85cff-6b8c-41de-b5dc-31b0e048dd26" (UID: "74a85cff-6b8c-41de-b5dc-31b0e048dd26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:25.363288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.363271 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a85cff-6b8c-41de-b5dc-31b0e048dd26-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.364782 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.364763 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a85cff-6b8c-41de-b5dc-31b0e048dd26-kube-api-access-9dqjw" pod "74a85cff-6b8c-41de-b5dc-31b0e048dd26" (UID: "74a85cff-6b8c-41de-b5dc-31b0e048dd26"). InnerVolumeSpecName "kube-api-access-9dqjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:25.464083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.464046 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dab119b-ecff-4560-a864-880e3b8c87da-kube-api-access-g27pv\" (UniqueName: \"kubernetes.io/projected/3dab119b-ecff-4560-a864-880e3b8c87da-kube-api-access-g27pv\") pod \"3dab119b-ecff-4560-a864-880e3b8c87da\" (UID: \"3dab119b-ecff-4560-a864-880e3b8c87da\") " Oct 31 00:35:25.464262 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.464227 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dab119b-ecff-4560-a864-880e3b8c87da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dab119b-ecff-4560-a864-880e3b8c87da-plugins-config-volume\") pod \"3dab119b-ecff-4560-a864-880e3b8c87da\" (UID: \"3dab119b-ecff-4560-a864-880e3b8c87da\") " Oct 31 00:35:25.464611 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.464572 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dab119b-ecff-4560-a864-880e3b8c87da-plugins-config-volume" pod "3dab119b-ecff-4560-a864-880e3b8c87da" (UID: "3dab119b-ecff-4560-a864-880e3b8c87da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:25.464786 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.464768 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dqjw\" (UniqueName: \"kubernetes.io/projected/74a85cff-6b8c-41de-b5dc-31b0e048dd26-kube-api-access-9dqjw\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.464854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.464791 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dab119b-ecff-4560-a864-880e3b8c87da-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.466259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.466240 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dab119b-ecff-4560-a864-880e3b8c87da-kube-api-access-g27pv" pod "3dab119b-ecff-4560-a864-880e3b8c87da" (UID: "3dab119b-ecff-4560-a864-880e3b8c87da"). InnerVolumeSpecName "kube-api-access-g27pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:25.566551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.566468 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g27pv\" (UniqueName: \"kubernetes.io/projected/3dab119b-ecff-4560-a864-880e3b8c87da-kube-api-access-g27pv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.749044 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.749005 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds"] Oct 31 00:35:25.753229 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:25.753204 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-7d684ff48f-ncsds"] Oct 31 00:35:26.336641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:26.336555 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv" Oct 31 00:35:26.502826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:26.502792 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv"] Oct 31 00:35:26.552382 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:26.552355 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-7c7b65b4c9-fswpv"] Oct 31 00:35:27.735854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:27.735821 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dab119b-ecff-4560-a864-880e3b8c87da" path="/var/lib/kubelet/pods/3dab119b-ecff-4560-a864-880e3b8c87da/volumes" Oct 31 00:35:27.736277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:27.736262 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a85cff-6b8c-41de-b5dc-31b0e048dd26" path="/var/lib/kubelet/pods/74a85cff-6b8c-41de-b5dc-31b0e048dd26/volumes" Oct 31 00:35:32.728640 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:32.728570 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:35:34.964896 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:34.964866 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj"] Oct 31 00:35:34.965281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:34.965083 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" containerID="cri-o://9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b" gracePeriod=30 Oct 31 00:35:36.689694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.689657 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4"] Oct 31 00:35:36.690307 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690292 4969 cpu_manager.go:414] "containerMap: removing container" podUID="1118a188-069d-4f54-b6a9-fc9cd626ef21" containerName="maas-api" Oct 31 00:35:36.690307 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690307 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1118a188-069d-4f54-b6a9-fc9cd626ef21" containerName="maas-api" Oct 31 00:35:36.690385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690317 4969 cpu_manager.go:414] "containerMap: removing container" podUID="3dab119b-ecff-4560-a864-880e3b8c87da" containerName="payload-pre-processing" Oct 31 00:35:36.690385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690323 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dab119b-ecff-4560-a864-880e3b8c87da" containerName="payload-pre-processing" Oct 31 00:35:36.690385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690342 4969 cpu_manager.go:414] "containerMap: removing container" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" Oct 31 00:35:36.690385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690346 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" Oct 31 00:35:36.690385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690354 4969 cpu_manager.go:414] "containerMap: removing container" podUID="74a85cff-6b8c-41de-b5dc-31b0e048dd26" containerName="payload-processing" Oct 31 00:35:36.690385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690358 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a85cff-6b8c-41de-b5dc-31b0e048dd26" containerName="payload-processing" Oct 31 00:35:36.690385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690371 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d6934960-0fb3-42d3-9506-586edae56ad0" containerName="payload-pre-processing" Oct 31 00:35:36.690385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690376 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6934960-0fb3-42d3-9506-586edae56ad0" containerName="payload-pre-processing" Oct 31 00:35:36.690385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690386 4969 cpu_manager.go:414] "containerMap: removing container" podUID="acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec" containerName="payload-pre-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690391 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec" containerName="payload-pre-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690402 4969 cpu_manager.go:414] "containerMap: removing container" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690407 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690421 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" containerName="test-curl-7504-5eab08" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690425 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" containerName="test-curl-7504-5eab08" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690434 4969 cpu_manager.go:414] "containerMap: removing container" podUID="5752d604-f290-4a9d-b73a-4d8852d02864" containerName="payload-pre-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690439 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5752d604-f290-4a9d-b73a-4d8852d02864" containerName="payload-pre-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690451 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d75a9303-d87f-4ba8-81e1-2235dcf7685b" containerName="payload-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690455 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75a9303-d87f-4ba8-81e1-2235dcf7685b" containerName="payload-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690556 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690564 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a85cff-6b8c-41de-b5dc-31b0e048dd26" containerName="payload-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690570 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6934960-0fb3-42d3-9506-586edae56ad0" containerName="payload-pre-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690578 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc2ac6f-fab4-4d04-bd1e-fa768c6c91ec" containerName="payload-pre-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690585 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e037e7f2-6470-4fa0-a1cc-1c741b8d33c7" containerName="test-curl-7504-5eab08" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690604 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="5752d604-f290-4a9d-b73a-4d8852d02864" containerName="payload-pre-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690611 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75a9303-d87f-4ba8-81e1-2235dcf7685b" containerName="payload-processing" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690618 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="1118a188-069d-4f54-b6a9-fc9cd626ef21" containerName="maas-api" Oct 31 00:35:36.690678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.690623 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dab119b-ecff-4560-a864-880e3b8c87da" containerName="payload-pre-processing" Oct 31 00:35:36.695436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.695418 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.750540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.750505 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f6a8a5-openshift-default-dockercfg-2mqhs\"" Oct 31 00:35:36.804872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.804834 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4"] Oct 31 00:35:36.823249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.823208 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-envoy\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.823421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.823268 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-podinfo\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.823421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.823406 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-certs\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.823559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.823448 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istiod-ca-cert\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.823559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.823490 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-socket\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.823759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.823669 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prrzn\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-kube-api-access-prrzn\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.823841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.823821 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-credential-socket\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.823904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.823877 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-data\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.823962 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.823906 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-token\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.924874 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.924836 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-credential-socket\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.924912 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-data\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.924940 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-token\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925002 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-envoy\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925030 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-podinfo\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925121 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-certs\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925149 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istiod-ca-cert\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925198 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-socket\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925280 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-credential-socket\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925303 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prrzn\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-kube-api-access-prrzn\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925609 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925563 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-socket\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.925857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925828 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-data\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.926047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925902 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-certs\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.926047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.925988 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istiod-ca-cert\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.927559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.927538 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-envoy\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.927962 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.927940 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-podinfo\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.986459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.986377 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-token\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:36.986459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:36.986427 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prrzn\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-kube-api-access-prrzn\") pod \"e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:37.009416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:37.009372 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:37.471155 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:35:37.471109 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbf9386_7d9d_4f0e_bbbd_da2a080cfdaa.slice/crio-4bd7f4cee3ff30f602d56f62a6afabccb05cb26774267743399ee115dc640cfc WatchSource:0}: Error finding container 4bd7f4cee3ff30f602d56f62a6afabccb05cb26774267743399ee115dc640cfc: Status 404 returned error can't find the container with id 4bd7f4cee3ff30f602d56f62a6afabccb05cb26774267743399ee115dc640cfc Oct 31 00:35:37.471496 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:37.471474 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4"] Oct 31 00:35:37.473587 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:37.473552 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:35:37.473676 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:37.473661 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:35:37.473715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:37.473696 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:35:38.435792 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:38.435753 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" event={"ID":"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa","Type":"ContainerStarted","Data":"9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3"} Oct 31 00:35:38.436196 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:38.435811 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" event={"ID":"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa","Type":"ContainerStarted","Data":"4bd7f4cee3ff30f602d56f62a6afabccb05cb26774267743399ee115dc640cfc"} Oct 31 00:35:38.612228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:38.612176 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" podStartSLOduration=2.612162857 podStartE2EDuration="2.612162857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:38.610875863 +0000 UTC m=+1971.086234289" watchObservedRunningTime="2026-10-31 00:35:38.612162857 +0000 UTC m=+1971.087521284" Oct 31 00:35:39.010498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:39.010461 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:39.015920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:39.015895 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:39.443868 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:39.443835 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:39.444750 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:39.444731 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:35:40.249915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.249857 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:35:40.378969 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.378935 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-podinfo\") pod \"dfdec536-f521-4f73-b2ef-8d421a1a688b\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " Oct 31 00:35:40.379115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.378985 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-token\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-token\") pod \"dfdec536-f521-4f73-b2ef-8d421a1a688b\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " Oct 31 00:35:40.379115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.379051 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-data\") pod \"dfdec536-f521-4f73-b2ef-8d421a1a688b\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " Oct 31 00:35:40.379115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.379093 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-kube-api-access-d5zrc\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-kube-api-access-d5zrc\") pod \"dfdec536-f521-4f73-b2ef-8d421a1a688b\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " Oct 31 00:35:40.379231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.379152 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-credential-socket\") pod \"dfdec536-f521-4f73-b2ef-8d421a1a688b\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " Oct 31 00:35:40.379287 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.379269 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-envoy\") pod \"dfdec536-f521-4f73-b2ef-8d421a1a688b\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " Oct 31 00:35:40.379343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.379322 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-socket\") pod \"dfdec536-f521-4f73-b2ef-8d421a1a688b\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " Oct 31 00:35:40.379430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.379396 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-certs\") pod \"dfdec536-f521-4f73-b2ef-8d421a1a688b\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " Oct 31 00:35:40.379498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.379451 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-data" pod "dfdec536-f521-4f73-b2ef-8d421a1a688b" (UID: "dfdec536-f521-4f73-b2ef-8d421a1a688b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:40.379498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.379461 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfdec536-f521-4f73-b2ef-8d421a1a688b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfdec536-f521-4f73-b2ef-8d421a1a688b-istiod-ca-cert\") pod \"dfdec536-f521-4f73-b2ef-8d421a1a688b\" (UID: \"dfdec536-f521-4f73-b2ef-8d421a1a688b\") " Oct 31 00:35:40.380128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.379956 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-certs" pod "dfdec536-f521-4f73-b2ef-8d421a1a688b" (UID: "dfdec536-f521-4f73-b2ef-8d421a1a688b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:40.380128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.380017 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfdec536-f521-4f73-b2ef-8d421a1a688b-istiod-ca-cert" pod "dfdec536-f521-4f73-b2ef-8d421a1a688b" (UID: "dfdec536-f521-4f73-b2ef-8d421a1a688b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:40.380314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.380139 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-credential-socket" pod "dfdec536-f521-4f73-b2ef-8d421a1a688b" (UID: "dfdec536-f521-4f73-b2ef-8d421a1a688b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:40.380314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.380158 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-socket" pod "dfdec536-f521-4f73-b2ef-8d421a1a688b" (UID: "dfdec536-f521-4f73-b2ef-8d421a1a688b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:40.380433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.380417 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.380498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.380441 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.380498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.380456 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfdec536-f521-4f73-b2ef-8d421a1a688b-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.380498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.380470 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.380498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.380483 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.381689 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.381614 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-token" pod "dfdec536-f521-4f73-b2ef-8d421a1a688b" (UID: "dfdec536-f521-4f73-b2ef-8d421a1a688b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:40.381794 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.381689 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-kube-api-access-d5zrc" pod "dfdec536-f521-4f73-b2ef-8d421a1a688b" (UID: "dfdec536-f521-4f73-b2ef-8d421a1a688b"). InnerVolumeSpecName "kube-api-access-d5zrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:40.381794 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.381728 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-podinfo" pod "dfdec536-f521-4f73-b2ef-8d421a1a688b" (UID: "dfdec536-f521-4f73-b2ef-8d421a1a688b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:35:40.381794 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.381732 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-envoy" pod "dfdec536-f521-4f73-b2ef-8d421a1a688b" (UID: "dfdec536-f521-4f73-b2ef-8d421a1a688b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:40.454242 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.454209 4969 generic.go:356] "Generic (PLEG): container finished" podID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerID="9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b" exitCode=0 Oct 31 00:35:40.454662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.454256 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" Oct 31 00:35:40.454662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.454285 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" event={"ID":"dfdec536-f521-4f73-b2ef-8d421a1a688b","Type":"ContainerDied","Data":"9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b"} Oct 31 00:35:40.454662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.454358 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj" event={"ID":"dfdec536-f521-4f73-b2ef-8d421a1a688b","Type":"ContainerDied","Data":"7a0b43bea1fcb9714a5b92223413f8c74a6641a1c836f7175a08260c7d15e4fb"} Oct 31 00:35:40.454662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.454381 4969 scope.go:122] "RemoveContainer" containerID="9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b" Oct 31 00:35:40.478954 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.473430 4969 scope.go:122] "RemoveContainer" containerID="dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939" Oct 31 00:35:40.482073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.482040 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5zrc\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-kube-api-access-d5zrc\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.482207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.482080 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.482207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.482097 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.482207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.482113 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfdec536-f521-4f73-b2ef-8d421a1a688b-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:40.487213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.487185 4969 scope.go:122] "RemoveContainer" containerID="9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b" Oct 31 00:35:40.487544 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:40.487511 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b\": container with ID starting with 9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b not found: ID does not exist" containerID="9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b" Oct 31 00:35:40.487652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.487548 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b"} err="failed to get container status \"9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b\": rpc error: code = NotFound desc = could not find container \"9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b\": container with ID starting with 9872dc2db4ac58a71e7e2d9401cda395da1394a9cfc35147d8e084ffab1e883b not found: ID does not exist" Oct 31 00:35:40.487652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.487575 4969 scope.go:122] "RemoveContainer" containerID="dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939" Oct 31 00:35:40.488015 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:40.487988 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939\": container with ID starting with dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939 not found: ID does not exist" containerID="dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939" Oct 31 00:35:40.488090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.488023 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939"} err="failed to get container status \"dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939\": rpc error: code = NotFound desc = could not find container \"dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939\": container with ID starting with dc6e6adee8d414cff29bec704847525e4057cf215094dc8a5b31025553f72939 not found: ID does not exist" Oct 31 00:35:40.623762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.623732 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj"] Oct 31 00:35:40.627665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:40.627631 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2ccdde-openshift-default-87bd7f9df-9xlfj"] Oct 31 00:35:41.736010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:41.735976 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" path="/var/lib/kubelet/pods/dfdec536-f521-4f73-b2ef-8d421a1a688b/volumes" Oct 31 00:35:45.196462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.196379 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5"] Oct 31 00:35:45.197071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.197056 4969 cpu_manager.go:414] "containerMap: removing container" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" Oct 31 00:35:45.197126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.197073 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" Oct 31 00:35:45.197126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.197112 4969 cpu_manager.go:414] "containerMap: removing container" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" Oct 31 00:35:45.197126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.197121 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" Oct 31 00:35:45.197245 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.197236 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" Oct 31 00:35:45.197281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.197249 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e6a1ab-f335-42fc-a645-b18f8d554ead" containerName="istio-proxy" Oct 31 00:35:45.197281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.197258 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfdec536-f521-4f73-b2ef-8d421a1a688b" containerName="istio-proxy" Oct 31 00:35:45.202276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.202248 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:45.257687 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.257659 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5"] Oct 31 00:35:45.264569 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.264536 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8d8e11-ad4a-4a68-abca-aa543c98d320-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5\" (UID: \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:45.264861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.264831 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jdr\" (UniqueName: \"kubernetes.io/projected/ee8d8e11-ad4a-4a68-abca-aa543c98d320-kube-api-access-x4jdr\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5\" (UID: \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:45.302550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.302525 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f6a8a5\"" Oct 31 00:35:45.370617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.366986 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jdr\" (UniqueName: \"kubernetes.io/projected/ee8d8e11-ad4a-4a68-abca-aa543c98d320-kube-api-access-x4jdr\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5\" (UID: \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:45.370617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.367163 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8d8e11-ad4a-4a68-abca-aa543c98d320-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5\" (UID: \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:45.370617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.368354 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8d8e11-ad4a-4a68-abca-aa543c98d320-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5\" (UID: \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:45.388354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.388317 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml"] Oct 31 00:35:45.393558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.393535 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:45.449686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.444697 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jdr\" (UniqueName: \"kubernetes.io/projected/ee8d8e11-ad4a-4a68-abca-aa543c98d320-kube-api-access-x4jdr\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5\" (UID: \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:45.449686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.447942 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f6a8a5-dockercfg-ncfsp\"" Oct 31 00:35:45.450379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.450354 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml"] Oct 31 00:35:45.468884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.468853 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctr4l\" (UniqueName: \"kubernetes.io/projected/edca4be9-ccd3-4b86-94b2-4a78545aa199-kube-api-access-ctr4l\") pod \"payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml\" (UID: \"edca4be9-ccd3-4b86-94b2-4a78545aa199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:45.469072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.468916 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca4be9-ccd3-4b86-94b2-4a78545aa199-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml\" (UID: \"edca4be9-ccd3-4b86-94b2-4a78545aa199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:45.516322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.516292 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:45.570545 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.570515 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctr4l\" (UniqueName: \"kubernetes.io/projected/edca4be9-ccd3-4b86-94b2-4a78545aa199-kube-api-access-ctr4l\") pod \"payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml\" (UID: \"edca4be9-ccd3-4b86-94b2-4a78545aa199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:45.570731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.570565 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca4be9-ccd3-4b86-94b2-4a78545aa199-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml\" (UID: \"edca4be9-ccd3-4b86-94b2-4a78545aa199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:45.571523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.571497 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca4be9-ccd3-4b86-94b2-4a78545aa199-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml\" (UID: \"edca4be9-ccd3-4b86-94b2-4a78545aa199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:45.633034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.633003 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctr4l\" (UniqueName: \"kubernetes.io/projected/edca4be9-ccd3-4b86-94b2-4a78545aa199-kube-api-access-ctr4l\") pod \"payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml\" (UID: \"edca4be9-ccd3-4b86-94b2-4a78545aa199\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:45.709304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.709224 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:45.959146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:45.959120 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5"] Oct 31 00:35:45.960117 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:35:45.960055 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8d8e11_ad4a_4a68_abca_aa543c98d320.slice/crio-2aad7a5529bf9f7884db9cdfba808d178058270a64d179a3e482c2de64e8bf40 WatchSource:0}: Error finding container 2aad7a5529bf9f7884db9cdfba808d178058270a64d179a3e482c2de64e8bf40: Status 404 returned error can't find the container with id 2aad7a5529bf9f7884db9cdfba808d178058270a64d179a3e482c2de64e8bf40 Oct 31 00:35:46.092704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:46.092675 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml"] Oct 31 00:35:46.093297 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:35:46.093271 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedca4be9_ccd3_4b86_94b2_4a78545aa199.slice/crio-80c1aa927e637b17ff923dca89603e2be549a5f73e63896cf7721bf06ed38817 WatchSource:0}: Error finding container 80c1aa927e637b17ff923dca89603e2be549a5f73e63896cf7721bf06ed38817: Status 404 returned error can't find the container with id 80c1aa927e637b17ff923dca89603e2be549a5f73e63896cf7721bf06ed38817 Oct 31 00:35:46.507448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:46.507402 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" event={"ID":"ee8d8e11-ad4a-4a68-abca-aa543c98d320","Type":"ContainerStarted","Data":"3aeb73511e4ee80a1dcd5c02e1bc044bbfc4826129c756f459290a9f674dd672"} Oct 31 00:35:46.507936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:46.507472 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:46.507936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:46.507495 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" event={"ID":"ee8d8e11-ad4a-4a68-abca-aa543c98d320","Type":"ContainerStarted","Data":"2aad7a5529bf9f7884db9cdfba808d178058270a64d179a3e482c2de64e8bf40"} Oct 31 00:35:46.509355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:46.509324 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" event={"ID":"edca4be9-ccd3-4b86-94b2-4a78545aa199","Type":"ContainerStarted","Data":"1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939"} Oct 31 00:35:46.509522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:46.509373 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" event={"ID":"edca4be9-ccd3-4b86-94b2-4a78545aa199","Type":"ContainerStarted","Data":"80c1aa927e637b17ff923dca89603e2be549a5f73e63896cf7721bf06ed38817"} Oct 31 00:35:46.509522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:46.509413 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:46.624744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:46.624695 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" podStartSLOduration=1.624680492 podStartE2EDuration="1.624680492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:46.623207263 +0000 UTC m=+1979.098565699" watchObservedRunningTime="2026-10-31 00:35:46.624680492 +0000 UTC m=+1979.100038914" Oct 31 00:35:46.741149 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:46.741084 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" podStartSLOduration=1.741065124 podStartE2EDuration="1.741065124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:46.738270957 +0000 UTC m=+1979.213629381" watchObservedRunningTime="2026-10-31 00:35:46.741065124 +0000 UTC m=+1979.216423547" Oct 31 00:35:48.940626 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:48.940566 4969 scope.go:122] "RemoveContainer" containerID="b85686e9e04744deda1cde25ed086de105e61965dae1f6115b3f3f3fc1c738fc" Oct 31 00:35:48.951685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:48.951660 4969 scope.go:122] "RemoveContainer" containerID="081d774a1a1d3123b4ef0b42458dfab5f7e95ee5c6104b46af6146b07bc213d8" Oct 31 00:35:49.945321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:49.945288 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb"] Oct 31 00:35:49.945745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:49.945504 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" containerID="cri-o://beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8" gracePeriod=30 Oct 31 00:35:49.965279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:49.965239 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7"] Oct 31 00:35:49.965559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:49.965514 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" containerID="cri-o://9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab" gracePeriod=30 Oct 31 00:35:55.254579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.254547 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:35:55.283252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.283228 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:35:55.410901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.410867 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-envoy\") pod \"9c719883-0029-4053-b988-c48f5248e09a\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " Oct 31 00:35:55.411086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.410922 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-certs\") pod \"9c719883-0029-4053-b988-c48f5248e09a\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " Oct 31 00:35:55.411086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.410989 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c719883-0029-4053-b988-c48f5248e09a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c719883-0029-4053-b988-c48f5248e09a-istio-podinfo\") pod \"9c719883-0029-4053-b988-c48f5248e09a\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " Oct 31 00:35:55.411086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411025 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c719883-0029-4053-b988-c48f5248e09a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c719883-0029-4053-b988-c48f5248e09a-istiod-ca-cert\") pod \"9c719883-0029-4053-b988-c48f5248e09a\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " Oct 31 00:35:55.411086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411055 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-istio-token\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-istio-token\") pod \"9c719883-0029-4053-b988-c48f5248e09a\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " Oct 31 00:35:55.411315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411089 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-data\") pod \"decc71b4-6554-457b-86c9-969fe79ba955\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " Oct 31 00:35:55.411315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411122 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-istio-token\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-istio-token\") pod \"decc71b4-6554-457b-86c9-969fe79ba955\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " Oct 31 00:35:55.411315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411158 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-socket\") pod \"9c719883-0029-4053-b988-c48f5248e09a\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " Oct 31 00:35:55.411315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411189 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-socket\") pod \"decc71b4-6554-457b-86c9-969fe79ba955\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " Oct 31 00:35:55.411315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411246 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-certs" pod "9c719883-0029-4053-b988-c48f5248e09a" (UID: "9c719883-0029-4053-b988-c48f5248e09a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.411315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411298 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-envoy\") pod \"decc71b4-6554-457b-86c9-969fe79ba955\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411412 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-credential-socket\") pod \"9c719883-0029-4053-b988-c48f5248e09a\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411476 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-data" pod "decc71b4-6554-457b-86c9-969fe79ba955" (UID: "decc71b4-6554-457b-86c9-969fe79ba955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411494 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-kube-api-access-v25tj\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-kube-api-access-v25tj\") pod \"decc71b4-6554-457b-86c9-969fe79ba955\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411530 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-credential-socket\") pod \"decc71b4-6554-457b-86c9-969fe79ba955\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411561 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-kube-api-access-kgmkg\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-kube-api-access-kgmkg\") pod \"9c719883-0029-4053-b988-c48f5248e09a\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411646 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/decc71b4-6554-457b-86c9-969fe79ba955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decc71b4-6554-457b-86c9-969fe79ba955-istio-podinfo\") pod \"decc71b4-6554-457b-86c9-969fe79ba955\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411674 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decc71b4-6554-457b-86c9-969fe79ba955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decc71b4-6554-457b-86c9-969fe79ba955-istiod-ca-cert\") pod \"decc71b4-6554-457b-86c9-969fe79ba955\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411714 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-data\") pod \"9c719883-0029-4053-b988-c48f5248e09a\" (UID: \"9c719883-0029-4053-b988-c48f5248e09a\") " Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411775 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-certs\") pod \"decc71b4-6554-457b-86c9-969fe79ba955\" (UID: \"decc71b4-6554-457b-86c9-969fe79ba955\") " Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.411809 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-socket" pod "decc71b4-6554-457b-86c9-969fe79ba955" (UID: "decc71b4-6554-457b-86c9-969fe79ba955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.412405 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-credential-socket" pod "decc71b4-6554-457b-86c9-969fe79ba955" (UID: "decc71b4-6554-457b-86c9-969fe79ba955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.412567 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.412954 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.412980 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.413055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.412996 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.417434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.414154 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-socket" pod "9c719883-0029-4053-b988-c48f5248e09a" (UID: "9c719883-0029-4053-b988-c48f5248e09a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.417434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.414544 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decc71b4-6554-457b-86c9-969fe79ba955-istiod-ca-cert" pod "decc71b4-6554-457b-86c9-969fe79ba955" (UID: "decc71b4-6554-457b-86c9-969fe79ba955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:55.417434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.414986 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-credential-socket" pod "9c719883-0029-4053-b988-c48f5248e09a" (UID: "9c719883-0029-4053-b988-c48f5248e09a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.417434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.415121 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-istio-token" pod "decc71b4-6554-457b-86c9-969fe79ba955" (UID: "decc71b4-6554-457b-86c9-969fe79ba955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:55.417434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.415259 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-certs" pod "decc71b4-6554-457b-86c9-969fe79ba955" (UID: "decc71b4-6554-457b-86c9-969fe79ba955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.417434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.415318 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-data" pod "9c719883-0029-4053-b988-c48f5248e09a" (UID: "9c719883-0029-4053-b988-c48f5248e09a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.417434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.415621 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c719883-0029-4053-b988-c48f5248e09a-istiod-ca-cert" pod "9c719883-0029-4053-b988-c48f5248e09a" (UID: "9c719883-0029-4053-b988-c48f5248e09a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:55.417434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.415675 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-kube-api-access-kgmkg" pod "9c719883-0029-4053-b988-c48f5248e09a" (UID: "9c719883-0029-4053-b988-c48f5248e09a"). InnerVolumeSpecName "kube-api-access-kgmkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:55.417841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.417511 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/decc71b4-6554-457b-86c9-969fe79ba955-istio-podinfo" pod "decc71b4-6554-457b-86c9-969fe79ba955" (UID: "decc71b4-6554-457b-86c9-969fe79ba955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:35:55.418984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.418957 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-istio-token" pod "9c719883-0029-4053-b988-c48f5248e09a" (UID: "9c719883-0029-4053-b988-c48f5248e09a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:55.419096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.419073 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-envoy" pod "9c719883-0029-4053-b988-c48f5248e09a" (UID: "9c719883-0029-4053-b988-c48f5248e09a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.419722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.419683 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-kube-api-access-v25tj" pod "decc71b4-6554-457b-86c9-969fe79ba955" (UID: "decc71b4-6554-457b-86c9-969fe79ba955"). InnerVolumeSpecName "kube-api-access-v25tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:55.419815 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.419774 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-envoy" pod "decc71b4-6554-457b-86c9-969fe79ba955" (UID: "decc71b4-6554-457b-86c9-969fe79ba955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:55.420711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.420682 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c719883-0029-4053-b988-c48f5248e09a-istio-podinfo" pod "9c719883-0029-4053-b988-c48f5248e09a" (UID: "9c719883-0029-4053-b988-c48f5248e09a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:35:55.514491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514409 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v25tj\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-kube-api-access-v25tj\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514438 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgmkg\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-kube-api-access-kgmkg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514449 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/decc71b4-6554-457b-86c9-969fe79ba955-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514458 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/decc71b4-6554-457b-86c9-969fe79ba955-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514467 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514475 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514483 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514491 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c719883-0029-4053-b988-c48f5248e09a-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514498 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c719883-0029-4053-b988-c48f5248e09a-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514506 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c719883-0029-4053-b988-c48f5248e09a-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514514 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decc71b4-6554-457b-86c9-969fe79ba955-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514523 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514531 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/decc71b4-6554-457b-86c9-969fe79ba955-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.514928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.514538 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c719883-0029-4053-b988-c48f5248e09a-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:35:55.596172 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.596128 4969 generic.go:356] "Generic (PLEG): container finished" podID="9c719883-0029-4053-b988-c48f5248e09a" containerID="beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8" exitCode=0 Oct 31 00:35:55.596356 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.596179 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" Oct 31 00:35:55.596356 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.596195 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" event={"ID":"9c719883-0029-4053-b988-c48f5248e09a","Type":"ContainerDied","Data":"beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8"} Oct 31 00:35:55.596356 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.596258 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb" event={"ID":"9c719883-0029-4053-b988-c48f5248e09a","Type":"ContainerDied","Data":"423b3c20e455d24ccbff7be0a7c1f0c7f476a77e698a362beb60906fbc26e7b2"} Oct 31 00:35:55.596356 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.596288 4969 scope.go:122] "RemoveContainer" containerID="beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8" Oct 31 00:35:55.598158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.598133 4969 generic.go:356] "Generic (PLEG): container finished" podID="decc71b4-6554-457b-86c9-969fe79ba955" containerID="9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab" exitCode=0 Oct 31 00:35:55.598304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.598185 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" Oct 31 00:35:55.598304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.598246 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" event={"ID":"decc71b4-6554-457b-86c9-969fe79ba955","Type":"ContainerDied","Data":"9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab"} Oct 31 00:35:55.598304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.598286 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7" event={"ID":"decc71b4-6554-457b-86c9-969fe79ba955","Type":"ContainerDied","Data":"64ab076fe64afb6f5d2979b07bdcadc71578e656be6ba2189ee4e8ef4dec2373"} Oct 31 00:35:55.611351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.611201 4969 scope.go:122] "RemoveContainer" containerID="beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8" Oct 31 00:35:55.611539 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:55.611518 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8\": container with ID starting with beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8 not found: ID does not exist" containerID="beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8" Oct 31 00:35:55.612656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.611548 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8"} err="failed to get container status \"beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8\": rpc error: code = NotFound desc = could not find container \"beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8\": container with ID starting with beef431b754aa96a9f9e6cd8816cb76861fa6595c3ccd86722b3014ef6d887b8 not found: ID does not exist" Oct 31 00:35:55.612656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.611566 4969 scope.go:122] "RemoveContainer" containerID="9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab" Oct 31 00:35:55.625390 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.625364 4969 scope.go:122] "RemoveContainer" containerID="9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab" Oct 31 00:35:55.625774 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:35:55.625744 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab\": container with ID starting with 9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab not found: ID does not exist" containerID="9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab" Oct 31 00:35:55.625865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.625784 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab"} err="failed to get container status \"9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab\": rpc error: code = NotFound desc = could not find container \"9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab\": container with ID starting with 9b12def1dec7b2418ce89086646e8de53c3bed7db2bbca901ecdf234fd5384ab not found: ID does not exist" Oct 31 00:35:55.769099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.769013 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7"] Oct 31 00:35:55.774811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.774786 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-jgxm7"] Oct 31 00:35:55.891190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.891157 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb"] Oct 31 00:35:55.898465 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:55.898436 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-977vb"] Oct 31 00:35:57.524111 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:57.524085 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:35:57.524906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:57.524886 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:35:57.735736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:57.735700 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c719883-0029-4053-b988-c48f5248e09a" path="/var/lib/kubelet/pods/9c719883-0029-4053-b988-c48f5248e09a/volumes" Oct 31 00:35:57.736232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:35:57.736218 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decc71b4-6554-457b-86c9-969fe79ba955" path="/var/lib/kubelet/pods/decc71b4-6554-457b-86c9-969fe79ba955/volumes" Oct 31 00:36:02.729673 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:02.729629 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:36:03.048256 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:03.048170 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh"] Oct 31 00:36:03.098099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:03.098066 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed9b59b3-vg2mh"] Oct 31 00:36:03.747451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:03.747402 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1e92a8d-da40-4da2-b84d-4ebc228fc5d7" path="/var/lib/kubelet/pods/b1e92a8d-da40-4da2-b84d-4ebc228fc5d7/volumes" Oct 31 00:36:05.093405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.093368 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4"] Oct 31 00:36:05.094033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.094019 4969 cpu_manager.go:414] "containerMap: removing container" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" Oct 31 00:36:05.094091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.094035 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" Oct 31 00:36:05.094091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.094052 4969 cpu_manager.go:414] "containerMap: removing container" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" Oct 31 00:36:05.094091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.094057 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" Oct 31 00:36:05.094182 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.094166 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c719883-0029-4053-b988-c48f5248e09a" containerName="istio-proxy" Oct 31 00:36:05.094219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.094184 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="decc71b4-6554-457b-86c9-969fe79ba955" containerName="istio-proxy" Oct 31 00:36:05.099435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.099409 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:05.152710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.152677 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2c7172-53f4-42f9-8966-6aa877920ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4\" (UID: \"3a2c7172-53f4-42f9-8966-6aa877920ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:05.152913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.152732 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7bq\" (UniqueName: \"kubernetes.io/projected/3a2c7172-53f4-42f9-8966-6aa877920ae2-kube-api-access-rl7bq\") pod \"payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4\" (UID: \"3a2c7172-53f4-42f9-8966-6aa877920ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:05.155896 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.155871 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4"] Oct 31 00:36:05.200783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.200757 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-70babe\"" Oct 31 00:36:05.223088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.223064 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-70babe-dockercfg-hcmtj\"" Oct 31 00:36:05.259425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.259392 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2c7172-53f4-42f9-8966-6aa877920ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4\" (UID: \"3a2c7172-53f4-42f9-8966-6aa877920ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:05.259620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.259436 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7bq\" (UniqueName: \"kubernetes.io/projected/3a2c7172-53f4-42f9-8966-6aa877920ae2-kube-api-access-rl7bq\") pod \"payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4\" (UID: \"3a2c7172-53f4-42f9-8966-6aa877920ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:05.260136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.260113 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2c7172-53f4-42f9-8966-6aa877920ae2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4\" (UID: \"3a2c7172-53f4-42f9-8966-6aa877920ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:05.320693 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.320657 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7bq\" (UniqueName: \"kubernetes.io/projected/3a2c7172-53f4-42f9-8966-6aa877920ae2-kube-api-access-rl7bq\") pod \"payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4\" (UID: \"3a2c7172-53f4-42f9-8966-6aa877920ae2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:05.414470 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.414435 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:05.798556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.798523 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4"] Oct 31 00:36:05.798965 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:36:05.798943 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2c7172_53f4_42f9_8966_6aa877920ae2.slice/crio-d65ee17e7465011e7822fcd3bc9bdcf075d2da4131d43eee0699b65570e3b180 WatchSource:0}: Error finding container d65ee17e7465011e7822fcd3bc9bdcf075d2da4131d43eee0699b65570e3b180: Status 404 returned error can't find the container with id d65ee17e7465011e7822fcd3bc9bdcf075d2da4131d43eee0699b65570e3b180 Oct 31 00:36:05.882739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.882701 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl"] Oct 31 00:36:05.887086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.887068 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:05.942922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.942887 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2"] Oct 31 00:36:05.943139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.943113 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" containerID="cri-o://de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd" gracePeriod=30 Oct 31 00:36:05.954249 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.954221 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl"] Oct 31 00:36:05.969991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.969959 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl\" (UID: \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:05.970194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.970173 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdxnm\" (UniqueName: \"kubernetes.io/projected/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-kube-api-access-gdxnm\") pod \"payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl\" (UID: \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:05.978848 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.978828 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74"] Oct 31 00:36:05.979016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.978997 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" containerID="cri-o://40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9" gracePeriod=30 Oct 31 00:36:05.994947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.992946 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b"] Oct 31 00:36:05.994947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.993353 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" containerID="cri-o://069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f" gracePeriod=30 Oct 31 00:36:05.998606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:05.998567 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd"] Oct 31 00:36:06.001878 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.001846 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" containerID="cri-o://e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693" gracePeriod=30 Oct 31 00:36:06.071364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.071287 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl\" (UID: \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:06.071503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.071414 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdxnm\" (UniqueName: \"kubernetes.io/projected/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-kube-api-access-gdxnm\") pod \"payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl\" (UID: \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:06.072091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.072072 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl\" (UID: \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:06.167615 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.165009 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdxnm\" (UniqueName: \"kubernetes.io/projected/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-kube-api-access-gdxnm\") pod \"payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl\" (UID: \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:06.214538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.214502 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:06.603727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.603694 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl"] Oct 31 00:36:06.603964 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:36:06.603933 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ecefa7_43bc_4a70_950e_2e28d2dc6e43.slice/crio-7fee01e7580c166f2fdae057d2b1fcf498b96736ece83ba37d2b7d636ac3c590 WatchSource:0}: Error finding container 7fee01e7580c166f2fdae057d2b1fcf498b96736ece83ba37d2b7d636ac3c590: Status 404 returned error can't find the container with id 7fee01e7580c166f2fdae057d2b1fcf498b96736ece83ba37d2b7d636ac3c590 Oct 31 00:36:06.689873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.689843 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" event={"ID":"3a2c7172-53f4-42f9-8966-6aa877920ae2","Type":"ContainerStarted","Data":"371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e"} Oct 31 00:36:06.689991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.689904 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" event={"ID":"3a2c7172-53f4-42f9-8966-6aa877920ae2","Type":"ContainerStarted","Data":"d65ee17e7465011e7822fcd3bc9bdcf075d2da4131d43eee0699b65570e3b180"} Oct 31 00:36:06.689991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.689938 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:06.691776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.691751 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" event={"ID":"70ecefa7-43bc-4a70-950e-2e28d2dc6e43","Type":"ContainerStarted","Data":"7fee01e7580c166f2fdae057d2b1fcf498b96736ece83ba37d2b7d636ac3c590"} Oct 31 00:36:06.808558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:06.808503 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" podStartSLOduration=1.808489284 podStartE2EDuration="1.808489284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:06.805168519 +0000 UTC m=+1999.280526944" watchObservedRunningTime="2026-10-31 00:36:06.808489284 +0000 UTC m=+1999.283847706" Oct 31 00:36:07.156964 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:07.156930 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg"] Oct 31 00:36:07.157208 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:07.157183 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" containerID="cri-o://b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6" gracePeriod=30 Oct 31 00:36:07.701449 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:07.701412 4969 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:07.702002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:07.701468 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:07.703799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:07.703758 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" event={"ID":"70ecefa7-43bc-4a70-950e-2e28d2dc6e43","Type":"ContainerStarted","Data":"16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61"} Oct 31 00:36:07.703960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:07.703851 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:07.831212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:07.831154 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" podStartSLOduration=2.83113444 podStartE2EDuration="2.83113444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:07.824197677 +0000 UTC m=+2000.299556100" watchObservedRunningTime="2026-10-31 00:36:07.83113444 +0000 UTC m=+2000.306492864" Oct 31 00:36:08.119583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:08.119541 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:08.119782 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:08.119627 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:08.250635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:08.249849 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:08.250635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:08.249910 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:08.407512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:08.407428 4969 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:08.407512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:08.407495 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:11.314818 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.314786 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:36:11.386273 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.386245 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:36:11.389729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.389708 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:36:11.393989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.393972 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:36:11.451551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451514 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-token\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-token\") pod \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " Oct 31 00:36:11.451764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451575 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-data\") pod \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " Oct 31 00:36:11.451764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451655 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-envoy\") pod \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " Oct 31 00:36:11.451764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451693 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-data\") pod \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " Oct 31 00:36:11.451764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451723 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-credential-socket\") pod \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " Oct 31 00:36:11.451764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451756 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-socket\") pod \"a0b7814a-5fef-47e6-83a9-b1c511303a04\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " Oct 31 00:36:11.452086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451798 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-socket\") pod \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " Oct 31 00:36:11.452086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451848 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istiod-ca-cert\") pod \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " Oct 31 00:36:11.452086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451873 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-credential-socket\") pod \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " Oct 31 00:36:11.452086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451948 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-kube-api-access-njlnq\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-kube-api-access-njlnq\") pod \"ea391266-6c79-49c5-a25c-e8c48cf07f55\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " Oct 31 00:36:11.452086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451949 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-data" pod "ec837b0a-64cd-48c8-9cb0-bebedfd18d26" (UID: "ec837b0a-64cd-48c8-9cb0-bebedfd18d26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.452086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.451978 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-envoy\") pod \"ea391266-6c79-49c5-a25c-e8c48cf07f55\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " Oct 31 00:36:11.452086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452039 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-token\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-token\") pod \"ea391266-6c79-49c5-a25c-e8c48cf07f55\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " Oct 31 00:36:11.452086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452081 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-kube-api-access-lb56f\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-kube-api-access-lb56f\") pod \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452116 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-podinfo\") pod \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452168 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-socket" pod "a0b7814a-5fef-47e6-83a9-b1c511303a04" (UID: "a0b7814a-5fef-47e6-83a9-b1c511303a04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452191 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-kube-api-access-fsgqh\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-kube-api-access-fsgqh\") pod \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452222 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-envoy\") pod \"a0b7814a-5fef-47e6-83a9-b1c511303a04\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452250 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-certs\") pod \"ea391266-6c79-49c5-a25c-e8c48cf07f55\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452300 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istiod-ca-cert\") pod \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452333 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-data\") pod \"ea391266-6c79-49c5-a25c-e8c48cf07f55\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452370 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-token\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-token\") pod \"a0b7814a-5fef-47e6-83a9-b1c511303a04\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452415 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-credential-socket\") pod \"ea391266-6c79-49c5-a25c-e8c48cf07f55\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " Oct 31 00:36:11.452486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452456 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-socket\") pod \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452498 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea391266-6c79-49c5-a25c-e8c48cf07f55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391266-6c79-49c5-a25c-e8c48cf07f55-istiod-ca-cert\") pod \"ea391266-6c79-49c5-a25c-e8c48cf07f55\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452501 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-data" pod "025a7d0c-7431-4b83-961e-46dceb7ebc1e" (UID: "025a7d0c-7431-4b83-961e-46dceb7ebc1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452528 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-podinfo\") pod \"a0b7814a-5fef-47e6-83a9-b1c511303a04\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452562 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b7814a-5fef-47e6-83a9-b1c511303a04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b7814a-5fef-47e6-83a9-b1c511303a04-istiod-ca-cert\") pod \"a0b7814a-5fef-47e6-83a9-b1c511303a04\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452614 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-certs\") pod \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452665 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-podinfo\") pod \"ea391266-6c79-49c5-a25c-e8c48cf07f55\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452698 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-socket\") pod \"ea391266-6c79-49c5-a25c-e8c48cf07f55\" (UID: \"ea391266-6c79-49c5-a25c-e8c48cf07f55\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452771 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-kube-api-access-xvkzd\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-kube-api-access-xvkzd\") pod \"a0b7814a-5fef-47e6-83a9-b1c511303a04\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452827 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-certs\") pod \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452870 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-credential-socket\") pod \"a0b7814a-5fef-47e6-83a9-b1c511303a04\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452901 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-certs\") pod \"a0b7814a-5fef-47e6-83a9-b1c511303a04\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452937 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-credential-socket" pod "ea391266-6c79-49c5-a25c-e8c48cf07f55" (UID: "ea391266-6c79-49c5-a25c-e8c48cf07f55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452945 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-envoy\") pod \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\" (UID: \"ec837b0a-64cd-48c8-9cb0-bebedfd18d26\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.452982 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-token\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-token\") pod \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.453019 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-data\") pod \"a0b7814a-5fef-47e6-83a9-b1c511303a04\" (UID: \"a0b7814a-5fef-47e6-83a9-b1c511303a04\") " Oct 31 00:36:11.453058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.453054 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-podinfo\") pod \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\" (UID: \"025a7d0c-7431-4b83-961e-46dceb7ebc1e\") " Oct 31 00:36:11.454076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.453188 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-socket" pod "ec837b0a-64cd-48c8-9cb0-bebedfd18d26" (UID: "ec837b0a-64cd-48c8-9cb0-bebedfd18d26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.454076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.453539 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-socket" pod "025a7d0c-7431-4b83-961e-46dceb7ebc1e" (UID: "025a7d0c-7431-4b83-961e-46dceb7ebc1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.454076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.453874 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-certs" pod "ea391266-6c79-49c5-a25c-e8c48cf07f55" (UID: "ea391266-6c79-49c5-a25c-e8c48cf07f55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.454076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.453952 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.454076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.453978 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.454076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.453993 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.454076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454007 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.454076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454021 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.454076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454033 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.454538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454172 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istiod-ca-cert" pod "ec837b0a-64cd-48c8-9cb0-bebedfd18d26" (UID: "ec837b0a-64cd-48c8-9cb0-bebedfd18d26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.454538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454175 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istiod-ca-cert" pod "025a7d0c-7431-4b83-961e-46dceb7ebc1e" (UID: "025a7d0c-7431-4b83-961e-46dceb7ebc1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.454538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454229 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-credential-socket" pod "ec837b0a-64cd-48c8-9cb0-bebedfd18d26" (UID: "ec837b0a-64cd-48c8-9cb0-bebedfd18d26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.454538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454252 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b7814a-5fef-47e6-83a9-b1c511303a04-istiod-ca-cert" pod "a0b7814a-5fef-47e6-83a9-b1c511303a04" (UID: "a0b7814a-5fef-47e6-83a9-b1c511303a04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.455283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454618 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-data" pod "ea391266-6c79-49c5-a25c-e8c48cf07f55" (UID: "ea391266-6c79-49c5-a25c-e8c48cf07f55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.455283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454669 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-credential-socket" pod "025a7d0c-7431-4b83-961e-46dceb7ebc1e" (UID: "025a7d0c-7431-4b83-961e-46dceb7ebc1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.455283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.454866 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-certs" pod "ec837b0a-64cd-48c8-9cb0-bebedfd18d26" (UID: "ec837b0a-64cd-48c8-9cb0-bebedfd18d26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.455448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.455412 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-socket" pod "ea391266-6c79-49c5-a25c-e8c48cf07f55" (UID: "ea391266-6c79-49c5-a25c-e8c48cf07f55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.455511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.455494 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea391266-6c79-49c5-a25c-e8c48cf07f55-istiod-ca-cert" pod "ea391266-6c79-49c5-a25c-e8c48cf07f55" (UID: "ea391266-6c79-49c5-a25c-e8c48cf07f55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:11.455814 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.455788 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-certs" pod "025a7d0c-7431-4b83-961e-46dceb7ebc1e" (UID: "025a7d0c-7431-4b83-961e-46dceb7ebc1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.456935 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.456758 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-certs" pod "a0b7814a-5fef-47e6-83a9-b1c511303a04" (UID: "a0b7814a-5fef-47e6-83a9-b1c511303a04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.456935 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.456837 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-envoy" pod "a0b7814a-5fef-47e6-83a9-b1c511303a04" (UID: "a0b7814a-5fef-47e6-83a9-b1c511303a04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.457495 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.457470 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-kube-api-access-fsgqh" pod "025a7d0c-7431-4b83-961e-46dceb7ebc1e" (UID: "025a7d0c-7431-4b83-961e-46dceb7ebc1e"). InnerVolumeSpecName "kube-api-access-fsgqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.458558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.458293 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-data" pod "a0b7814a-5fef-47e6-83a9-b1c511303a04" (UID: "a0b7814a-5fef-47e6-83a9-b1c511303a04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.458558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.458516 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-token" pod "a0b7814a-5fef-47e6-83a9-b1c511303a04" (UID: "a0b7814a-5fef-47e6-83a9-b1c511303a04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.458855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.458821 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-credential-socket" pod "a0b7814a-5fef-47e6-83a9-b1c511303a04" (UID: "a0b7814a-5fef-47e6-83a9-b1c511303a04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.459347 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.459323 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-podinfo" pod "ec837b0a-64cd-48c8-9cb0-bebedfd18d26" (UID: "ec837b0a-64cd-48c8-9cb0-bebedfd18d26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:11.460309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.460272 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-kube-api-access-lb56f" pod "ec837b0a-64cd-48c8-9cb0-bebedfd18d26" (UID: "ec837b0a-64cd-48c8-9cb0-bebedfd18d26"). InnerVolumeSpecName "kube-api-access-lb56f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.461027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.460627 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-token" pod "ea391266-6c79-49c5-a25c-e8c48cf07f55" (UID: "ea391266-6c79-49c5-a25c-e8c48cf07f55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.461027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.460691 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-envoy" pod "ea391266-6c79-49c5-a25c-e8c48cf07f55" (UID: "ea391266-6c79-49c5-a25c-e8c48cf07f55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.461027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.460951 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-kube-api-access-njlnq" pod "ea391266-6c79-49c5-a25c-e8c48cf07f55" (UID: "ea391266-6c79-49c5-a25c-e8c48cf07f55"). InnerVolumeSpecName "kube-api-access-njlnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.461027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.461000 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-podinfo" pod "ea391266-6c79-49c5-a25c-e8c48cf07f55" (UID: "ea391266-6c79-49c5-a25c-e8c48cf07f55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:11.461986 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.461965 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-kube-api-access-xvkzd" pod "a0b7814a-5fef-47e6-83a9-b1c511303a04" (UID: "a0b7814a-5fef-47e6-83a9-b1c511303a04"). InnerVolumeSpecName "kube-api-access-xvkzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.462086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.461987 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-token" pod "ec837b0a-64cd-48c8-9cb0-bebedfd18d26" (UID: "ec837b0a-64cd-48c8-9cb0-bebedfd18d26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.462198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.462172 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-envoy" pod "025a7d0c-7431-4b83-961e-46dceb7ebc1e" (UID: "025a7d0c-7431-4b83-961e-46dceb7ebc1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.462291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.462176 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-token" pod "025a7d0c-7431-4b83-961e-46dceb7ebc1e" (UID: "025a7d0c-7431-4b83-961e-46dceb7ebc1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:11.462384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.462366 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-podinfo" pod "a0b7814a-5fef-47e6-83a9-b1c511303a04" (UID: "a0b7814a-5fef-47e6-83a9-b1c511303a04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:11.463247 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.463223 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-podinfo" pod "025a7d0c-7431-4b83-961e-46dceb7ebc1e" (UID: "025a7d0c-7431-4b83-961e-46dceb7ebc1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:11.463407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.463391 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-envoy" pod "ec837b0a-64cd-48c8-9cb0-bebedfd18d26" (UID: "ec837b0a-64cd-48c8-9cb0-bebedfd18d26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:11.555695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555580 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.555695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555630 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.555695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555644 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.555695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555656 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea391266-6c79-49c5-a25c-e8c48cf07f55-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.555695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555669 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.555695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555682 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b7814a-5fef-47e6-83a9-b1c511303a04-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.555695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555694 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555707 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555719 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555731 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvkzd\" (UniqueName: \"kubernetes.io/projected/a0b7814a-5fef-47e6-83a9-b1c511303a04-kube-api-access-xvkzd\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555745 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555756 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555769 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555780 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555792 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555804 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555816 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555828 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555839 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555853 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a7d0c-7431-4b83-961e-46dceb7ebc1e-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555865 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a7d0c-7431-4b83-961e-46dceb7ebc1e-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555877 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555892 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njlnq\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-kube-api-access-njlnq\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555906 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555925 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea391266-6c79-49c5-a25c-e8c48cf07f55-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555940 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb56f\" (UniqueName: \"kubernetes.io/projected/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-kube-api-access-lb56f\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555955 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec837b0a-64cd-48c8-9cb0-bebedfd18d26-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555969 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsgqh\" (UniqueName: \"kubernetes.io/projected/025a7d0c-7431-4b83-961e-46dceb7ebc1e-kube-api-access-fsgqh\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555984 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b7814a-5fef-47e6-83a9-b1c511303a04-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.556119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.555997 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea391266-6c79-49c5-a25c-e8c48cf07f55-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:11.747903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.747853 4969 generic.go:356] "Generic (PLEG): container finished" podID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerID="de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd" exitCode=0 Oct 31 00:36:11.748101 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.747919 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" Oct 31 00:36:11.748101 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.748044 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" event={"ID":"ea391266-6c79-49c5-a25c-e8c48cf07f55","Type":"ContainerDied","Data":"de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd"} Oct 31 00:36:11.748101 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.748090 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2" event={"ID":"ea391266-6c79-49c5-a25c-e8c48cf07f55","Type":"ContainerDied","Data":"82423d3e4f843d5f0a3eaa6b8cfd8fe98f09c214ddda76b3cca4aa1eb59b5222"} Oct 31 00:36:11.748270 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.748117 4969 scope.go:122] "RemoveContainer" containerID="de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd" Oct 31 00:36:11.759355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.759146 4969 generic.go:356] "Generic (PLEG): container finished" podID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerID="069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f" exitCode=0 Oct 31 00:36:11.759799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.759774 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" event={"ID":"ec837b0a-64cd-48c8-9cb0-bebedfd18d26","Type":"ContainerDied","Data":"069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f"} Oct 31 00:36:11.759920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.759874 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" event={"ID":"ec837b0a-64cd-48c8-9cb0-bebedfd18d26","Type":"ContainerDied","Data":"1015f0c2b4e35152370e097fd7ef81908d7b7db80cb7a3b00c1e8f485fc3a5d7"} Oct 31 00:36:11.759989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.759945 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b" Oct 31 00:36:11.783175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.772625 4969 scope.go:122] "RemoveContainer" containerID="de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd" Oct 31 00:36:11.783175 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:11.779047 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd\": container with ID starting with de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd not found: ID does not exist" containerID="de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd" Oct 31 00:36:11.783175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.779086 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd"} err="failed to get container status \"de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd\": rpc error: code = NotFound desc = could not find container \"de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd\": container with ID starting with de278f3983e8d67a7ac6cfbc143314ef4a51c423877612cc136804794bf585cd not found: ID does not exist" Oct 31 00:36:11.783175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.779110 4969 scope.go:122] "RemoveContainer" containerID="069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f" Oct 31 00:36:11.783175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.781411 4969 generic.go:356] "Generic (PLEG): container finished" podID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerID="40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9" exitCode=0 Oct 31 00:36:11.783175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.782101 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" Oct 31 00:36:11.783175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.782130 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" event={"ID":"025a7d0c-7431-4b83-961e-46dceb7ebc1e","Type":"ContainerDied","Data":"40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9"} Oct 31 00:36:11.783175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.782187 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74" event={"ID":"025a7d0c-7431-4b83-961e-46dceb7ebc1e","Type":"ContainerDied","Data":"cac5b1b1ad4d09117be09bf50ea4bac03005580f6aaa75525918b3b2142bc616"} Oct 31 00:36:11.785174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.785146 4969 generic.go:356] "Generic (PLEG): container finished" podID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerID="e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693" exitCode=0 Oct 31 00:36:11.785292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.785234 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" event={"ID":"a0b7814a-5fef-47e6-83a9-b1c511303a04","Type":"ContainerDied","Data":"e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693"} Oct 31 00:36:11.785292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.785266 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" event={"ID":"a0b7814a-5fef-47e6-83a9-b1c511303a04","Type":"ContainerDied","Data":"baf65187c089efb18d903f97b68e9c1556a120cb44057337b643be924769c6b9"} Oct 31 00:36:11.785448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.785338 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd" Oct 31 00:36:11.792643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.792623 4969 scope.go:122] "RemoveContainer" containerID="069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f" Oct 31 00:36:11.792915 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:11.792898 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f\": container with ID starting with 069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f not found: ID does not exist" containerID="069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f" Oct 31 00:36:11.792970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.792921 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f"} err="failed to get container status \"069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f\": rpc error: code = NotFound desc = could not find container \"069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f\": container with ID starting with 069678d012554b92d2835451629037e95eb9b329261056bace540c0dab98e89f not found: ID does not exist" Oct 31 00:36:11.792970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.792934 4969 scope.go:122] "RemoveContainer" containerID="40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9" Oct 31 00:36:11.803025 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.803008 4969 scope.go:122] "RemoveContainer" containerID="40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9" Oct 31 00:36:11.803303 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:11.803281 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9\": container with ID starting with 40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9 not found: ID does not exist" containerID="40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9" Oct 31 00:36:11.803378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.803356 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9"} err="failed to get container status \"40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9\": rpc error: code = NotFound desc = could not find container \"40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9\": container with ID starting with 40f303d02de1aa2795ee11477cb55006c235265a6db62d5c0303a343f73565e9 not found: ID does not exist" Oct 31 00:36:11.803426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.803383 4969 scope.go:122] "RemoveContainer" containerID="e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693" Oct 31 00:36:11.820771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.820741 4969 scope.go:122] "RemoveContainer" containerID="fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21" Oct 31 00:36:11.844546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.844521 4969 scope.go:122] "RemoveContainer" containerID="e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693" Oct 31 00:36:11.844998 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:11.844972 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693\": container with ID starting with e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693 not found: ID does not exist" containerID="e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693" Oct 31 00:36:11.845098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.845004 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693"} err="failed to get container status \"e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693\": rpc error: code = NotFound desc = could not find container \"e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693\": container with ID starting with e92eb68a1256cf1304e7a885eed98c706ebd60b8f9b3f7a3ed95debe53ac8693 not found: ID does not exist" Oct 31 00:36:11.845098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.845027 4969 scope.go:122] "RemoveContainer" containerID="fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21" Oct 31 00:36:11.845363 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:11.845331 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21\": container with ID starting with fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21 not found: ID does not exist" containerID="fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21" Oct 31 00:36:11.845439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.845369 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21"} err="failed to get container status \"fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21\": rpc error: code = NotFound desc = could not find container \"fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21\": container with ID starting with fe32cd370f9e4935ad9ed55263a776c9e0b32e65d92e66b345e421ecf9774e21 not found: ID does not exist" Oct 31 00:36:11.964694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.964661 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b"] Oct 31 00:36:11.968304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:11.968281 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-bct8b"] Oct 31 00:36:12.078317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.078230 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd"] Oct 31 00:36:12.081824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.081803 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-gjnsd"] Oct 31 00:36:12.195960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.195926 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74"] Oct 31 00:36:12.198940 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.198910 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-4hd74"] Oct 31 00:36:12.314619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.314558 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2"] Oct 31 00:36:12.319911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.319879 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-tqsg2"] Oct 31 00:36:12.429719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.429690 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:36:12.572016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.571981 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-envoy\") pod \"2d1047f6-6bc7-422a-9633-04fcfb13920c\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " Oct 31 00:36:12.572214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.572102 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-token\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-token\") pod \"2d1047f6-6bc7-422a-9633-04fcfb13920c\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " Oct 31 00:36:12.572214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.572132 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-podinfo\") pod \"2d1047f6-6bc7-422a-9633-04fcfb13920c\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " Oct 31 00:36:12.572214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.572186 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-kube-api-access-s9nll\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-kube-api-access-s9nll\") pod \"2d1047f6-6bc7-422a-9633-04fcfb13920c\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " Oct 31 00:36:12.572398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.572292 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-socket\") pod \"2d1047f6-6bc7-422a-9633-04fcfb13920c\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " Oct 31 00:36:12.572398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.572334 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-certs\") pod \"2d1047f6-6bc7-422a-9633-04fcfb13920c\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " Oct 31 00:36:12.572398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.572374 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-credential-socket\") pod \"2d1047f6-6bc7-422a-9633-04fcfb13920c\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " Oct 31 00:36:12.572560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.572408 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1047f6-6bc7-422a-9633-04fcfb13920c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1047f6-6bc7-422a-9633-04fcfb13920c-istiod-ca-cert\") pod \"2d1047f6-6bc7-422a-9633-04fcfb13920c\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " Oct 31 00:36:12.572560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.572514 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-data\") pod \"2d1047f6-6bc7-422a-9633-04fcfb13920c\" (UID: \"2d1047f6-6bc7-422a-9633-04fcfb13920c\") " Oct 31 00:36:12.573234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.573187 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-credential-socket" pod "2d1047f6-6bc7-422a-9633-04fcfb13920c" (UID: "2d1047f6-6bc7-422a-9633-04fcfb13920c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.573392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.573348 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-certs" pod "2d1047f6-6bc7-422a-9633-04fcfb13920c" (UID: "2d1047f6-6bc7-422a-9633-04fcfb13920c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.573392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.573365 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-data" pod "2d1047f6-6bc7-422a-9633-04fcfb13920c" (UID: "2d1047f6-6bc7-422a-9633-04fcfb13920c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.573542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.573508 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1047f6-6bc7-422a-9633-04fcfb13920c-istiod-ca-cert" pod "2d1047f6-6bc7-422a-9633-04fcfb13920c" (UID: "2d1047f6-6bc7-422a-9633-04fcfb13920c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:12.573850 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.573797 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-socket" pod "2d1047f6-6bc7-422a-9633-04fcfb13920c" (UID: "2d1047f6-6bc7-422a-9633-04fcfb13920c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.576748 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.576720 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-envoy" pod "2d1047f6-6bc7-422a-9633-04fcfb13920c" (UID: "2d1047f6-6bc7-422a-9633-04fcfb13920c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:12.576748 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.576739 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-podinfo" pod "2d1047f6-6bc7-422a-9633-04fcfb13920c" (UID: "2d1047f6-6bc7-422a-9633-04fcfb13920c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:12.576895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.576765 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-kube-api-access-s9nll" pod "2d1047f6-6bc7-422a-9633-04fcfb13920c" (UID: "2d1047f6-6bc7-422a-9633-04fcfb13920c"). InnerVolumeSpecName "kube-api-access-s9nll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:12.576895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.576812 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-token" pod "2d1047f6-6bc7-422a-9633-04fcfb13920c" (UID: "2d1047f6-6bc7-422a-9633-04fcfb13920c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:12.674613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.674562 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.674613 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.674611 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.674840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.674626 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.674840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.674638 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nll\" (UniqueName: \"kubernetes.io/projected/2d1047f6-6bc7-422a-9633-04fcfb13920c-kube-api-access-s9nll\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.674840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.674652 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.674840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.674663 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.674840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.674676 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.674840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.674689 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1047f6-6bc7-422a-9633-04fcfb13920c-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.674840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.674701 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1047f6-6bc7-422a-9633-04fcfb13920c-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:12.797219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.797182 4969 generic.go:356] "Generic (PLEG): container finished" podID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerID="b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6" exitCode=0 Oct 31 00:36:12.797409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.797214 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" event={"ID":"2d1047f6-6bc7-422a-9633-04fcfb13920c","Type":"ContainerDied","Data":"b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6"} Oct 31 00:36:12.797409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.797234 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" Oct 31 00:36:12.797409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.797267 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg" event={"ID":"2d1047f6-6bc7-422a-9633-04fcfb13920c","Type":"ContainerDied","Data":"f59b8cf9d91792f5989e25773c93fe17efa29f788813f4e6e18182e10bb21852"} Oct 31 00:36:12.797409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.797291 4969 scope.go:122] "RemoveContainer" containerID="b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6" Oct 31 00:36:12.815563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.815540 4969 scope.go:122] "RemoveContainer" containerID="b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6" Oct 31 00:36:12.815924 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:12.815898 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6\": container with ID starting with b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6 not found: ID does not exist" containerID="b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6" Oct 31 00:36:12.816008 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.815934 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6"} err="failed to get container status \"b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6\": rpc error: code = NotFound desc = could not find container \"b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6\": container with ID starting with b14a08b6c713fe8705f25873dc00bbf7d24be0ca19d4a80ab1946bd478f1efc6 not found: ID does not exist" Oct 31 00:36:12.968299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.968262 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg"] Oct 31 00:36:12.974526 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:12.974487 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-qrcjg"] Oct 31 00:36:13.735566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:13.735531 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" path="/var/lib/kubelet/pods/025a7d0c-7431-4b83-961e-46dceb7ebc1e/volumes" Oct 31 00:36:13.736090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:13.736072 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" path="/var/lib/kubelet/pods/2d1047f6-6bc7-422a-9633-04fcfb13920c/volumes" Oct 31 00:36:13.736505 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:13.736492 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" path="/var/lib/kubelet/pods/a0b7814a-5fef-47e6-83a9-b1c511303a04/volumes" Oct 31 00:36:13.737298 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:13.737283 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" path="/var/lib/kubelet/pods/ea391266-6c79-49c5-a25c-e8c48cf07f55/volumes" Oct 31 00:36:13.737763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:13.737749 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" path="/var/lib/kubelet/pods/ec837b0a-64cd-48c8-9cb0-bebedfd18d26/volumes" Oct 31 00:36:16.303002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.302963 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt"] Oct 31 00:36:16.303568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303554 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" Oct 31 00:36:16.303667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303569 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" Oct 31 00:36:16.303667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303604 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" Oct 31 00:36:16.303667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303610 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" Oct 31 00:36:16.303667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303621 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" Oct 31 00:36:16.303667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303625 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" Oct 31 00:36:16.303667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303645 4969 cpu_manager.go:414] "containerMap: removing container" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" Oct 31 00:36:16.303667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303650 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" Oct 31 00:36:16.303667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303662 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" Oct 31 00:36:16.303667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303666 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" Oct 31 00:36:16.303955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303685 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" Oct 31 00:36:16.303955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303689 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" Oct 31 00:36:16.303955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303774 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" Oct 31 00:36:16.303955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303782 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1047f6-6bc7-422a-9633-04fcfb13920c" containerName="istio-proxy" Oct 31 00:36:16.303955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303791 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec837b0a-64cd-48c8-9cb0-bebedfd18d26" containerName="istio-proxy" Oct 31 00:36:16.303955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303800 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="025a7d0c-7431-4b83-961e-46dceb7ebc1e" containerName="istio-proxy" Oct 31 00:36:16.303955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.303807 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea391266-6c79-49c5-a25c-e8c48cf07f55" containerName="istio-proxy" Oct 31 00:36:16.308765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.308747 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:16.367673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.367637 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt"] Oct 31 00:36:16.414622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.410755 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-752be8\"" Oct 31 00:36:16.414622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.410992 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-752be8-dockercfg-vs46p\"" Oct 31 00:36:16.428697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.423464 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt\" (UID: \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:16.428697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.423759 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rtmt\" (UniqueName: \"kubernetes.io/projected/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-kube-api-access-4rtmt\") pod \"payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt\" (UID: \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:16.524721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.524685 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt\" (UID: \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:16.524910 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.524822 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rtmt\" (UniqueName: \"kubernetes.io/projected/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-kube-api-access-4rtmt\") pod \"payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt\" (UID: \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:16.525505 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.525484 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt\" (UID: \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:16.585800 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.585733 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rtmt\" (UniqueName: \"kubernetes.io/projected/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-kube-api-access-4rtmt\") pod \"payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt\" (UID: \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:16.623772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:16.623740 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:17.016492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:17.016462 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt"] Oct 31 00:36:17.016938 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:36:17.016911 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66768b27_3dd7_4b05_a1e7_e936dccb7e7c.slice/crio-3b94a50cb4888bbd8eef27a9181fbef15f0a35bdd9bc33267e3b65f5f3a0c00b WatchSource:0}: Error finding container 3b94a50cb4888bbd8eef27a9181fbef15f0a35bdd9bc33267e3b65f5f3a0c00b: Status 404 returned error can't find the container with id 3b94a50cb4888bbd8eef27a9181fbef15f0a35bdd9bc33267e3b65f5f3a0c00b Oct 31 00:36:17.705777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:17.705747 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:36:17.854706 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:17.854665 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" event={"ID":"66768b27-3dd7-4b05-a1e7-e936dccb7e7c","Type":"ContainerStarted","Data":"44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e"} Oct 31 00:36:17.855012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:17.854737 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" event={"ID":"66768b27-3dd7-4b05-a1e7-e936dccb7e7c","Type":"ContainerStarted","Data":"3b94a50cb4888bbd8eef27a9181fbef15f0a35bdd9bc33267e3b65f5f3a0c00b"} Oct 31 00:36:17.855510 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:17.855483 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:17.993729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:17.993559 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" podStartSLOduration=1.993541752 podStartE2EDuration="1.993541752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:17.992734027 +0000 UTC m=+2010.468092466" watchObservedRunningTime="2026-10-31 00:36:17.993541752 +0000 UTC m=+2010.468900179" Oct 31 00:36:18.715288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:18.715253 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:36:25.392753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:25.392715 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp"] Oct 31 00:36:25.393147 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:25.392961 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" containerID="cri-o://2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec" gracePeriod=30 Oct 31 00:36:29.093157 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:29.093124 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl_70ecefa7-43bc-4a70-950e-2e28d2dc6e43/payload-processing/0.log" Oct 31 00:36:29.874109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:29.874083 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:36:30.668927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.668866 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:36:30.727644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.727613 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-kube-api-access-jp75h\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-kube-api-access-jp75h\") pod \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " Oct 31 00:36:30.727831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.727662 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-certs\") pod \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " Oct 31 00:36:30.727831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.727687 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-socket\") pod \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " Oct 31 00:36:30.727831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.727773 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istiod-ca-cert\") pod \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " Oct 31 00:36:30.728001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.727968 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-certs" pod "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" (UID: "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:30.728083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.728007 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-socket" pod "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" (UID: "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:30.728221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.728199 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istiod-ca-cert" pod "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" (UID: "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:30.728330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.728264 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.728330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.728278 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.730125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.730101 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-kube-api-access-jp75h" pod "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" (UID: "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611"). InnerVolumeSpecName "kube-api-access-jp75h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:30.829026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.828994 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-data\") pod \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " Oct 31 00:36:30.829206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.829039 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-podinfo\") pod \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " Oct 31 00:36:30.829206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.829060 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-token\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-token\") pod \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " Oct 31 00:36:30.829206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.829084 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-envoy\") pod \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " Oct 31 00:36:30.829206 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.829199 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-credential-socket\") pod \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\" (UID: \"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611\") " Oct 31 00:36:30.829427 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.829308 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-data" pod "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" (UID: "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:30.829500 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.829466 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-credential-socket" pod "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" (UID: "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:30.830076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.830054 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.830134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.830086 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.830134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.830101 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.830134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.830115 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp75h\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-kube-api-access-jp75h\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.831504 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.831482 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-podinfo" pod "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" (UID: "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:30.831504 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.831491 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-token" pod "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" (UID: "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:30.831641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.831511 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-envoy" pod "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" (UID: "48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:30.930974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.930898 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.930974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.930932 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.930974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.930944 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:36:30.959829 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.959797 4969 generic.go:356] "Generic (PLEG): container finished" podID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerID="2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec" exitCode=0 Oct 31 00:36:30.960011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.959849 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" Oct 31 00:36:30.960011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.959874 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" event={"ID":"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611","Type":"ContainerDied","Data":"2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec"} Oct 31 00:36:30.960011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.959928 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp" event={"ID":"48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611","Type":"ContainerDied","Data":"9d77e3fd57dbcb64dd7982f4bfe9e13f1f2ca413e5a98687ef2bdfd76b18ee05"} Oct 31 00:36:30.960011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.959950 4969 scope.go:122] "RemoveContainer" containerID="2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec" Oct 31 00:36:30.971317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.971296 4969 scope.go:122] "RemoveContainer" containerID="501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002" Oct 31 00:36:30.983530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.983491 4969 scope.go:122] "RemoveContainer" containerID="2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec" Oct 31 00:36:30.984135 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:30.984095 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec\": container with ID starting with 2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec not found: ID does not exist" containerID="2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec" Oct 31 00:36:30.984243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.984137 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec"} err="failed to get container status \"2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec\": rpc error: code = NotFound desc = could not find container \"2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec\": container with ID starting with 2573ff0ee9405a86b5d9f0186f2d38075913c1ab41af15be58e87801ad423dec not found: ID does not exist" Oct 31 00:36:30.984243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.984166 4969 scope.go:122] "RemoveContainer" containerID="501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002" Oct 31 00:36:30.984493 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:30.984473 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002\": container with ID starting with 501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002 not found: ID does not exist" containerID="501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002" Oct 31 00:36:30.984556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:30.984500 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002"} err="failed to get container status \"501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002\": rpc error: code = NotFound desc = could not find container \"501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002\": container with ID starting with 501d82f4a6a22d1b6f1ebcc5154071f81d571321c4fb1bf38aaea960a7b95002 not found: ID does not exist" Oct 31 00:36:31.128009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:31.127974 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp"] Oct 31 00:36:31.134485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:31.134458 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3e7872-openshift-default-5d6b959b4f-l77bp"] Oct 31 00:36:31.735712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:31.735681 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" path="/var/lib/kubelet/pods/48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611/volumes" Oct 31 00:36:32.728349 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:36:32.728315 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:36:42.120368 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.120334 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9"] Oct 31 00:36:42.121056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.121039 4969 cpu_manager.go:414] "containerMap: removing container" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" Oct 31 00:36:42.121104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.121059 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" Oct 31 00:36:42.121104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.121086 4969 cpu_manager.go:414] "containerMap: removing container" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" Oct 31 00:36:42.121104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.121091 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" Oct 31 00:36:42.121230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.121220 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b7814a-5fef-47e6-83a9-b1c511303a04" containerName="istio-proxy" Oct 31 00:36:42.121264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.121243 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" Oct 31 00:36:42.121264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.121256 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a5d113-45e6-4ff3-9ac6-a2c4a3d1a611" containerName="istio-proxy" Oct 31 00:36:42.124615 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.124576 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:36:42.186826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.186792 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9\" (UID: \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:36:42.187329 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.186965 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6nj\" (UniqueName: \"kubernetes.io/projected/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-kube-api-access-6n6nj\") pod \"payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9\" (UID: \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:36:42.226584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.226555 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-97fc39-dockercfg-td9t8\"" Oct 31 00:36:42.226584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.226562 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-97fc39\"" Oct 31 00:36:42.230477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.230452 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9"] Oct 31 00:36:42.287981 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.287951 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6nj\" (UniqueName: \"kubernetes.io/projected/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-kube-api-access-6n6nj\") pod \"payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9\" (UID: \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:36:42.288145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.288092 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9\" (UID: \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:36:42.288873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.288843 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9\" (UID: \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:36:42.348927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.348899 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6nj\" (UniqueName: \"kubernetes.io/projected/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-kube-api-access-6n6nj\") pod \"payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9\" (UID: \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:36:42.440395 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.440332 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:36:42.820528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:42.820496 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9"] Oct 31 00:36:42.821005 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:36:42.820983 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a6bf7b_bf37_488d_8ffe_1f2655fd7457.slice/crio-d158ff6f9784b0956f8e8034264aa8d3dae23ce4abf81e080299680dc67cf18e WatchSource:0}: Error finding container d158ff6f9784b0956f8e8034264aa8d3dae23ce4abf81e080299680dc67cf18e: Status 404 returned error can't find the container with id d158ff6f9784b0956f8e8034264aa8d3dae23ce4abf81e080299680dc67cf18e Oct 31 00:36:43.100200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:43.100118 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" event={"ID":"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457","Type":"ContainerStarted","Data":"0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b"} Oct 31 00:36:43.100350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:43.100305 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:36:43.100350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:43.100341 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" event={"ID":"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457","Type":"ContainerStarted","Data":"d158ff6f9784b0956f8e8034264aa8d3dae23ce4abf81e080299680dc67cf18e"} Oct 31 00:36:43.219327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:43.219273 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" podStartSLOduration=1.219258744 podStartE2EDuration="1.219258744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:43.215517152 +0000 UTC m=+2035.690875598" watchObservedRunningTime="2026-10-31 00:36:43.219258744 +0000 UTC m=+2035.694617167" Oct 31 00:36:49.010861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:49.010801 4969 scope.go:122] "RemoveContainer" containerID="a268c087003b5f15f3e4744e37f97c8e2e3fe5e698400801d2c310529f3b8280" Oct 31 00:36:54.112712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:36:54.112683 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:37:02.729408 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:37:02.729375 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:37:08.207881 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.207841 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq"] Oct 31 00:37:08.214179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.214156 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.268434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.268401 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq"] Oct 31 00:37:08.313052 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.313011 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-752be8-serving-cert\"" Oct 31 00:37:08.327086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.327054 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235cae5-9e24-46ef-879a-4c88ff2220cb-openshift-service-ca\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.327245 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.327107 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-metrics-certs\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.327245 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.327210 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.327358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.327338 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfnhg\" (UniqueName: \"kubernetes.io/projected/1235cae5-9e24-46ef-879a-4c88ff2220cb-kube-api-access-cfnhg\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.429099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.429061 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfnhg\" (UniqueName: \"kubernetes.io/projected/1235cae5-9e24-46ef-879a-4c88ff2220cb-kube-api-access-cfnhg\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.429277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.429200 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235cae5-9e24-46ef-879a-4c88ff2220cb-openshift-service-ca\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.429277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.429254 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-metrics-certs\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.429351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.429304 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.429462 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:37:08.429448 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-serving-cert: secret "maas-api-e2e-api-a-752be8-serving-cert" not found Oct 31 00:37:08.429526 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:37:08.429517 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls podName:1235cae5-9e24-46ef-879a-4c88ff2220cb nodeName:}" failed. No retries permitted until 2026-10-31 00:37:08.929495348 +0000 UTC m=+2061.404853756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls") pod "maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" (UID: "1235cae5-9e24-46ef-879a-4c88ff2220cb") : secret "maas-api-e2e-api-a-752be8-serving-cert" not found Oct 31 00:37:08.429936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.429902 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235cae5-9e24-46ef-879a-4c88ff2220cb-openshift-service-ca\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.432267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.432239 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-metrics-certs\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.511318 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.511238 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfnhg\" (UniqueName: \"kubernetes.io/projected/1235cae5-9e24-46ef-879a-4c88ff2220cb-kube-api-access-cfnhg\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.934867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.934837 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:08.937549 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:08.937520 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls\") pod \"maas-api-e2e-api-a-752be8-644dbd4459-4jzfq\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:09.129213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:09.129181 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:09.530283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:09.530248 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq"] Oct 31 00:37:09.531454 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:37:09.531427 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1235cae5_9e24_46ef_879a_4c88ff2220cb.slice/crio-f95067d49b6bea3e2a7ac301651996213bd2c1806cb39e5f85076252c3d253d5 WatchSource:0}: Error finding container f95067d49b6bea3e2a7ac301651996213bd2c1806cb39e5f85076252c3d253d5: Status 404 returned error can't find the container with id f95067d49b6bea3e2a7ac301651996213bd2c1806cb39e5f85076252c3d253d5 Oct 31 00:37:09.533106 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:09.533046 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:37:10.342002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:10.341935 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" event={"ID":"1235cae5-9e24-46ef-879a-4c88ff2220cb","Type":"ContainerStarted","Data":"f95067d49b6bea3e2a7ac301651996213bd2c1806cb39e5f85076252c3d253d5"} Oct 31 00:37:11.351252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:11.351214 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" event={"ID":"1235cae5-9e24-46ef-879a-4c88ff2220cb","Type":"ContainerStarted","Data":"46e60a74c197b19816e179c1eaa101bed33bcdc164f1c7308dc946093ab30c7f"} Oct 31 00:37:11.351736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:11.351287 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:11.509383 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:11.509329 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" podStartSLOduration=2.589075973 podStartE2EDuration="3.509313795s" totalImagesPullingTime="920.237822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:11.506005975 +0000 UTC m=+2063.981364419" watchObservedRunningTime="2026-10-31 00:37:11.509313795 +0000 UTC m=+2063.984672237" Oct 31 00:37:15.383184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:15.383151 4969 generic.go:356] "Generic (PLEG): container finished" podID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerID="9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3" exitCode=0 Oct 31 00:37:15.383665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:15.383220 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" event={"ID":"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa","Type":"ContainerDied","Data":"9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3"} Oct 31 00:37:15.383665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:15.383659 4969 scope.go:122] "RemoveContainer" containerID="9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3" Oct 31 00:37:15.385138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:15.385114 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:37:15.385223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:15.385177 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:37:15.385223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:15.385208 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:37:16.408360 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:16.408322 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" event={"ID":"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa","Type":"ContainerStarted","Data":"5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad"} Oct 31 00:37:17.010468 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:17.010435 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:37:17.010791 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:17.010763 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:37:17.010858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:17.010819 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 00:37:17.365137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:17.365115 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:37:18.010188 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:18.010151 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:37:18.010655 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:18.010212 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 00:37:19.009774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:19.009742 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:37:19.009954 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:19.009794 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 00:37:20.010077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:20.010037 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:37:20.010486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:20.010108 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 00:37:21.010702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:21.010661 4969 patch_prober.go:29] interesting pod/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:37:21.011185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:21.010737 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 00:37:22.014396 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:22.014369 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:37:22.014875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:22.014770 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:37:22.015368 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:22.015352 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:37:32.727054 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:37:32.727010 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:37:39.350268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:39.350232 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml"] Oct 31 00:37:39.481996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:39.481954 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5"] Oct 31 00:37:44.351444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.351382 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" podUID="edca4be9-ccd3-4b86-94b2-4a78545aa199" containerName="payload-processing" containerID="cri-o://1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939" gracePeriod=25 Oct 31 00:37:44.483271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.483223 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" podUID="ee8d8e11-ad4a-4a68-abca-aa543c98d320" containerName="payload-pre-processing" containerID="cri-o://3aeb73511e4ee80a1dcd5c02e1bc044bbfc4826129c756f459290a9f674dd672" gracePeriod=25 Oct 31 00:37:44.641087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.640043 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:37:44.652142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.652110 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" event={"ID":"ee8d8e11-ad4a-4a68-abca-aa543c98d320","Type":"ContainerDied","Data":"3aeb73511e4ee80a1dcd5c02e1bc044bbfc4826129c756f459290a9f674dd672"} Oct 31 00:37:44.652305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.652186 4969 generic.go:356] "Generic (PLEG): container finished" podID="ee8d8e11-ad4a-4a68-abca-aa543c98d320" containerID="3aeb73511e4ee80a1dcd5c02e1bc044bbfc4826129c756f459290a9f674dd672" exitCode=0 Oct 31 00:37:44.656639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.653895 4969 generic.go:356] "Generic (PLEG): container finished" podID="edca4be9-ccd3-4b86-94b2-4a78545aa199" containerID="1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939" exitCode=0 Oct 31 00:37:44.656639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.653942 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" Oct 31 00:37:44.656639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.653971 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" event={"ID":"edca4be9-ccd3-4b86-94b2-4a78545aa199","Type":"ContainerDied","Data":"1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939"} Oct 31 00:37:44.656639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.654012 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml" event={"ID":"edca4be9-ccd3-4b86-94b2-4a78545aa199","Type":"ContainerDied","Data":"80c1aa927e637b17ff923dca89603e2be549a5f73e63896cf7721bf06ed38817"} Oct 31 00:37:44.656639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.654040 4969 scope.go:122] "RemoveContainer" containerID="1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939" Oct 31 00:37:44.665077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.665057 4969 scope.go:122] "RemoveContainer" containerID="1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939" Oct 31 00:37:44.665307 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:37:44.665285 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939\": container with ID starting with 1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939 not found: ID does not exist" containerID="1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939" Oct 31 00:37:44.665386 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.665319 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939"} err="failed to get container status \"1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939\": rpc error: code = NotFound desc = could not find container \"1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939\": container with ID starting with 1579936a1183c9e35de7e4ca5325d59c35ae311434545ee3c0d07e544622b939 not found: ID does not exist" Oct 31 00:37:44.705130 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.705099 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edca4be9-ccd3-4b86-94b2-4a78545aa199-kube-api-access-ctr4l\" (UniqueName: \"kubernetes.io/projected/edca4be9-ccd3-4b86-94b2-4a78545aa199-kube-api-access-ctr4l\") pod \"edca4be9-ccd3-4b86-94b2-4a78545aa199\" (UID: \"edca4be9-ccd3-4b86-94b2-4a78545aa199\") " Oct 31 00:37:44.705330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.705308 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edca4be9-ccd3-4b86-94b2-4a78545aa199-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca4be9-ccd3-4b86-94b2-4a78545aa199-plugins-config-volume\") pod \"edca4be9-ccd3-4b86-94b2-4a78545aa199\" (UID: \"edca4be9-ccd3-4b86-94b2-4a78545aa199\") " Oct 31 00:37:44.705995 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.705969 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edca4be9-ccd3-4b86-94b2-4a78545aa199-plugins-config-volume" pod "edca4be9-ccd3-4b86-94b2-4a78545aa199" (UID: "edca4be9-ccd3-4b86-94b2-4a78545aa199"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:44.708353 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.708326 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edca4be9-ccd3-4b86-94b2-4a78545aa199-kube-api-access-ctr4l" pod "edca4be9-ccd3-4b86-94b2-4a78545aa199" (UID: "edca4be9-ccd3-4b86-94b2-4a78545aa199"). InnerVolumeSpecName "kube-api-access-ctr4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:44.765400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.765366 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:37:44.807447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.807403 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca4be9-ccd3-4b86-94b2-4a78545aa199-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:37:44.807447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.807438 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctr4l\" (UniqueName: \"kubernetes.io/projected/edca4be9-ccd3-4b86-94b2-4a78545aa199-kube-api-access-ctr4l\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:37:44.908653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.908626 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8d8e11-ad4a-4a68-abca-aa543c98d320-kube-api-access-x4jdr\" (UniqueName: \"kubernetes.io/projected/ee8d8e11-ad4a-4a68-abca-aa543c98d320-kube-api-access-x4jdr\") pod \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\" (UID: \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\") " Oct 31 00:37:44.908875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.908753 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee8d8e11-ad4a-4a68-abca-aa543c98d320-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8d8e11-ad4a-4a68-abca-aa543c98d320-plugins-config-volume\") pod \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\" (UID: \"ee8d8e11-ad4a-4a68-abca-aa543c98d320\") " Oct 31 00:37:44.909195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.909171 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8d8e11-ad4a-4a68-abca-aa543c98d320-plugins-config-volume" pod "ee8d8e11-ad4a-4a68-abca-aa543c98d320" (UID: "ee8d8e11-ad4a-4a68-abca-aa543c98d320"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:44.909292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.909277 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8d8e11-ad4a-4a68-abca-aa543c98d320-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:37:44.911021 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:44.910998 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8d8e11-ad4a-4a68-abca-aa543c98d320-kube-api-access-x4jdr" pod "ee8d8e11-ad4a-4a68-abca-aa543c98d320" (UID: "ee8d8e11-ad4a-4a68-abca-aa543c98d320"). InnerVolumeSpecName "kube-api-access-x4jdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:45.010827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:45.010792 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jdr\" (UniqueName: \"kubernetes.io/projected/ee8d8e11-ad4a-4a68-abca-aa543c98d320-kube-api-access-x4jdr\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:37:45.083409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:45.083369 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml"] Oct 31 00:37:45.087379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:45.087352 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-b947fdc59-dfkml"] Oct 31 00:37:45.665148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:45.665076 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" Oct 31 00:37:45.665615 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:45.665076 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5" event={"ID":"ee8d8e11-ad4a-4a68-abca-aa543c98d320","Type":"ContainerDied","Data":"2aad7a5529bf9f7884db9cdfba808d178058270a64d179a3e482c2de64e8bf40"} Oct 31 00:37:45.665615 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:45.665209 4969 scope.go:122] "RemoveContainer" containerID="3aeb73511e4ee80a1dcd5c02e1bc044bbfc4826129c756f459290a9f674dd672" Oct 31 00:37:45.734933 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:45.734893 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edca4be9-ccd3-4b86-94b2-4a78545aa199" path="/var/lib/kubelet/pods/edca4be9-ccd3-4b86-94b2-4a78545aa199/volumes" Oct 31 00:37:45.844384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:45.844348 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5"] Oct 31 00:37:45.852662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:45.852624 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-858cb8766c-8vlf5"] Oct 31 00:37:47.735137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:47.735106 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee8d8e11-ad4a-4a68-abca-aa543c98d320" path="/var/lib/kubelet/pods/ee8d8e11-ad4a-4a68-abca-aa543c98d320/volumes" Oct 31 00:37:48.085433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:48.085358 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:37:48.092751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:48.092727 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:37:48.101074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:48.101050 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:37:48.109275 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:48.109257 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:37:49.082459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:49.082427 4969 scope.go:122] "RemoveContainer" containerID="e6e5fd20c60ac072d1acee056a3aa4586d09ab398573e7b16c98b52af0f45596" Oct 31 00:37:49.096701 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:37:49.096669 4969 scope.go:122] "RemoveContainer" containerID="c7147a8b8ccfe2443846aaae120c85c3351e303b899889f37b0eb91f03387a6f" Oct 31 00:38:02.729215 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:38:02.729130 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:38:20.138841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.138806 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg"] Oct 31 00:38:20.139368 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.139017 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" podUID="94deed1d-3a2c-40c7-bd4d-f171cc753481" containerName="maas-api" containerID="cri-o://2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8" gracePeriod=30 Oct 31 00:38:20.419274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.419248 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:38:20.564089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.564058 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94deed1d-3a2c-40c7-bd4d-f171cc753481-kube-api-access-zkjcx\" (UniqueName: \"kubernetes.io/projected/94deed1d-3a2c-40c7-bd4d-f171cc753481-kube-api-access-zkjcx\") pod \"94deed1d-3a2c-40c7-bd4d-f171cc753481\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " Oct 31 00:38:20.564269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.564176 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls\") pod \"94deed1d-3a2c-40c7-bd4d-f171cc753481\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " Oct 31 00:38:20.564269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.564211 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94deed1d-3a2c-40c7-bd4d-f171cc753481-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deed1d-3a2c-40c7-bd4d-f171cc753481-openshift-service-ca\") pod \"94deed1d-3a2c-40c7-bd4d-f171cc753481\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " Oct 31 00:38:20.564389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.564349 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-metrics-certs\") pod \"94deed1d-3a2c-40c7-bd4d-f171cc753481\" (UID: \"94deed1d-3a2c-40c7-bd4d-f171cc753481\") " Oct 31 00:38:20.564702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.564669 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94deed1d-3a2c-40c7-bd4d-f171cc753481-openshift-service-ca" pod "94deed1d-3a2c-40c7-bd4d-f171cc753481" (UID: "94deed1d-3a2c-40c7-bd4d-f171cc753481"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:20.565076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.565038 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deed1d-3a2c-40c7-bd4d-f171cc753481-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:20.566517 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.566488 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls" pod "94deed1d-3a2c-40c7-bd4d-f171cc753481" (UID: "94deed1d-3a2c-40c7-bd4d-f171cc753481"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:20.566657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.566528 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94deed1d-3a2c-40c7-bd4d-f171cc753481-kube-api-access-zkjcx" pod "94deed1d-3a2c-40c7-bd4d-f171cc753481" (UID: "94deed1d-3a2c-40c7-bd4d-f171cc753481"). InnerVolumeSpecName "kube-api-access-zkjcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:20.566657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.566635 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-metrics-certs" pod "94deed1d-3a2c-40c7-bd4d-f171cc753481" (UID: "94deed1d-3a2c-40c7-bd4d-f171cc753481"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:20.666462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.666357 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:20.666462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.666394 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkjcx\" (UniqueName: \"kubernetes.io/projected/94deed1d-3a2c-40c7-bd4d-f171cc753481-kube-api-access-zkjcx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:20.666462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.666410 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deed1d-3a2c-40c7-bd4d-f171cc753481-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:20.934509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.934426 4969 generic.go:356] "Generic (PLEG): container finished" podID="94deed1d-3a2c-40c7-bd4d-f171cc753481" containerID="2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8" exitCode=0 Oct 31 00:38:20.934683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.934535 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" Oct 31 00:38:20.934683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.934534 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" event={"ID":"94deed1d-3a2c-40c7-bd4d-f171cc753481","Type":"ContainerDied","Data":"2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8"} Oct 31 00:38:20.934683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.934668 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg" event={"ID":"94deed1d-3a2c-40c7-bd4d-f171cc753481","Type":"ContainerDied","Data":"238f2200611e09b133aa97ddf38e6270082c82f7af9865b04c76d15575947509"} Oct 31 00:38:20.934838 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.934697 4969 scope.go:122] "RemoveContainer" containerID="2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8" Oct 31 00:38:20.948578 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.948504 4969 scope.go:122] "RemoveContainer" containerID="2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8" Oct 31 00:38:20.948992 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:38:20.948966 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8\": container with ID starting with 2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8 not found: ID does not exist" containerID="2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8" Oct 31 00:38:20.949058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:20.949004 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8"} err="failed to get container status \"2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8\": rpc error: code = NotFound desc = could not find container \"2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8\": container with ID starting with 2b9abc4b898bd38bee5a259a8238aa4ed57ed15e54fb71ce2c4c4ce48302e8b8 not found: ID does not exist" Oct 31 00:38:21.105062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:21.105020 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg"] Oct 31 00:38:21.111254 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:21.111226 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2cda64-b7bb78795-ldqqg"] Oct 31 00:38:21.735051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:21.735019 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94deed1d-3a2c-40c7-bd4d-f171cc753481" path="/var/lib/kubelet/pods/94deed1d-3a2c-40c7-bd4d-f171cc753481/volumes" Oct 31 00:38:21.998937 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:21.998859 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm"] Oct 31 00:38:21.999102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:21.999066 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" containerID="cri-o://2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24" gracePeriod=30 Oct 31 00:38:22.049010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:22.048978 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr"] Oct 31 00:38:22.049201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:22.049182 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" containerID="cri-o://b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4" gracePeriod=30 Oct 31 00:38:24.038009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:24.037973 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-6bnvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:38:24.038414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:24.038026 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:38:24.790786 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:24.790753 4969 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-thkzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:38:24.790973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:24.790827 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:38:26.835902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.835865 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd"] Oct 31 00:38:26.836486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.836476 4969 cpu_manager.go:414] "containerMap: removing container" podUID="94deed1d-3a2c-40c7-bd4d-f171cc753481" containerName="maas-api" Oct 31 00:38:26.836486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.836487 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94deed1d-3a2c-40c7-bd4d-f171cc753481" containerName="maas-api" Oct 31 00:38:26.836634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.836520 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ee8d8e11-ad4a-4a68-abca-aa543c98d320" containerName="payload-pre-processing" Oct 31 00:38:26.836634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.836528 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee8d8e11-ad4a-4a68-abca-aa543c98d320" containerName="payload-pre-processing" Oct 31 00:38:26.836634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.836542 4969 cpu_manager.go:414] "containerMap: removing container" podUID="edca4be9-ccd3-4b86-94b2-4a78545aa199" containerName="payload-processing" Oct 31 00:38:26.836634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.836547 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edca4be9-ccd3-4b86-94b2-4a78545aa199" containerName="payload-processing" Oct 31 00:38:26.836855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.836678 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee8d8e11-ad4a-4a68-abca-aa543c98d320" containerName="payload-pre-processing" Oct 31 00:38:26.836855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.836690 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="edca4be9-ccd3-4b86-94b2-4a78545aa199" containerName="payload-processing" Oct 31 00:38:26.836855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.836700 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="94deed1d-3a2c-40c7-bd4d-f171cc753481" containerName="maas-api" Oct 31 00:38:26.841674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.841652 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:26.862084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.862060 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gg68\" (UniqueName: \"kubernetes.io/projected/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-kube-api-access-4gg68\") pod \"payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd\" (UID: \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:26.862240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.862220 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd\" (UID: \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:26.943375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.943334 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-726d5b-dockercfg-29h7l\"" Oct 31 00:38:26.943375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.943336 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-726d5b\"" Oct 31 00:38:26.947462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.947441 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd"] Oct 31 00:38:26.963820 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.963792 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd\" (UID: \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:26.963956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.963903 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gg68\" (UniqueName: \"kubernetes.io/projected/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-kube-api-access-4gg68\") pod \"payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd\" (UID: \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:26.964453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:26.964433 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd\" (UID: \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:27.025874 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.025849 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gg68\" (UniqueName: \"kubernetes.io/projected/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-kube-api-access-4gg68\") pod \"payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd\" (UID: \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:27.156693 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.156667 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:27.346501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.346467 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:38:27.369644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.369620 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-istio-token\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-istio-token\") pod \"61d42e79-429b-48a0-b53f-42a836067d2c\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " Oct 31 00:38:27.369806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.369672 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-data\") pod \"61d42e79-429b-48a0-b53f-42a836067d2c\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " Oct 31 00:38:27.369806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.369698 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-envoy\") pod \"61d42e79-429b-48a0-b53f-42a836067d2c\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " Oct 31 00:38:27.369806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.369766 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-kube-api-access-7sqnv\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-kube-api-access-7sqnv\") pod \"61d42e79-429b-48a0-b53f-42a836067d2c\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " Oct 31 00:38:27.369806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.369793 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-credential-socket\") pod \"61d42e79-429b-48a0-b53f-42a836067d2c\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " Oct 31 00:38:27.369990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.369944 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-socket\") pod \"61d42e79-429b-48a0-b53f-42a836067d2c\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " Oct 31 00:38:27.370030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.370008 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61d42e79-429b-48a0-b53f-42a836067d2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d42e79-429b-48a0-b53f-42a836067d2c-istio-podinfo\") pod \"61d42e79-429b-48a0-b53f-42a836067d2c\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " Oct 31 00:38:27.370084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.370039 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d42e79-429b-48a0-b53f-42a836067d2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d42e79-429b-48a0-b53f-42a836067d2c-istiod-ca-cert\") pod \"61d42e79-429b-48a0-b53f-42a836067d2c\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " Oct 31 00:38:27.370171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.370148 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-certs\") pod \"61d42e79-429b-48a0-b53f-42a836067d2c\" (UID: \"61d42e79-429b-48a0-b53f-42a836067d2c\") " Oct 31 00:38:27.370237 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.370212 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-data" pod "61d42e79-429b-48a0-b53f-42a836067d2c" (UID: "61d42e79-429b-48a0-b53f-42a836067d2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.370372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.370341 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-socket" pod "61d42e79-429b-48a0-b53f-42a836067d2c" (UID: "61d42e79-429b-48a0-b53f-42a836067d2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.370540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.370449 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-credential-socket" pod "61d42e79-429b-48a0-b53f-42a836067d2c" (UID: "61d42e79-429b-48a0-b53f-42a836067d2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.370764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.370555 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-certs" pod "61d42e79-429b-48a0-b53f-42a836067d2c" (UID: "61d42e79-429b-48a0-b53f-42a836067d2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.370885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.370855 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d42e79-429b-48a0-b53f-42a836067d2c-istiod-ca-cert" pod "61d42e79-429b-48a0-b53f-42a836067d2c" (UID: "61d42e79-429b-48a0-b53f-42a836067d2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:27.371043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.371017 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.371110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.371050 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.371110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.371066 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d42e79-429b-48a0-b53f-42a836067d2c-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.371110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.371080 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.371110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.371094 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.372263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.372238 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-envoy" pod "61d42e79-429b-48a0-b53f-42a836067d2c" (UID: "61d42e79-429b-48a0-b53f-42a836067d2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.372452 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.372426 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-istio-token" pod "61d42e79-429b-48a0-b53f-42a836067d2c" (UID: "61d42e79-429b-48a0-b53f-42a836067d2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:27.372452 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.372442 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-kube-api-access-7sqnv" pod "61d42e79-429b-48a0-b53f-42a836067d2c" (UID: "61d42e79-429b-48a0-b53f-42a836067d2c"). InnerVolumeSpecName "kube-api-access-7sqnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:27.372899 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.372876 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61d42e79-429b-48a0-b53f-42a836067d2c-istio-podinfo" pod "61d42e79-429b-48a0-b53f-42a836067d2c" (UID: "61d42e79-429b-48a0-b53f-42a836067d2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:38:27.384579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.384559 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:38:27.471692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.471663 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-podinfo\") pod \"2f6e4058-e548-4a56-9017-5b57dad353ce\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " Oct 31 00:38:27.471853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.471730 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-credential-socket\") pod \"2f6e4058-e548-4a56-9017-5b57dad353ce\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " Oct 31 00:38:27.471853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.471772 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-kube-api-access-rcqdf\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-kube-api-access-rcqdf\") pod \"2f6e4058-e548-4a56-9017-5b57dad353ce\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " Oct 31 00:38:27.471853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.471807 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-socket\") pod \"2f6e4058-e548-4a56-9017-5b57dad353ce\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " Oct 31 00:38:27.472022 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.471888 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6e4058-e548-4a56-9017-5b57dad353ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6e4058-e548-4a56-9017-5b57dad353ce-istiod-ca-cert\") pod \"2f6e4058-e548-4a56-9017-5b57dad353ce\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " Oct 31 00:38:27.472022 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.471933 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-envoy\") pod \"2f6e4058-e548-4a56-9017-5b57dad353ce\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " Oct 31 00:38:27.472022 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.471961 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-data\") pod \"2f6e4058-e548-4a56-9017-5b57dad353ce\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " Oct 31 00:38:27.472022 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.472002 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-token\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-token\") pod \"2f6e4058-e548-4a56-9017-5b57dad353ce\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " Oct 31 00:38:27.472214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.472044 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-certs\") pod \"2f6e4058-e548-4a56-9017-5b57dad353ce\" (UID: \"2f6e4058-e548-4a56-9017-5b57dad353ce\") " Oct 31 00:38:27.472214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.472062 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-socket" pod "2f6e4058-e548-4a56-9017-5b57dad353ce" (UID: "2f6e4058-e548-4a56-9017-5b57dad353ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.472214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.472076 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-credential-socket" pod "2f6e4058-e548-4a56-9017-5b57dad353ce" (UID: "2f6e4058-e548-4a56-9017-5b57dad353ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.472377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.472353 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-data" pod "2f6e4058-e548-4a56-9017-5b57dad353ce" (UID: "2f6e4058-e548-4a56-9017-5b57dad353ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473020 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-certs" pod "2f6e4058-e548-4a56-9017-5b57dad353ce" (UID: "2f6e4058-e548-4a56-9017-5b57dad353ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473361 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6e4058-e548-4a56-9017-5b57dad353ce-istiod-ca-cert" pod "2f6e4058-e548-4a56-9017-5b57dad353ce" (UID: "2f6e4058-e548-4a56-9017-5b57dad353ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473525 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473543 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sqnv\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-kube-api-access-7sqnv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473556 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473568 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d42e79-429b-48a0-b53f-42a836067d2c-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473581 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6e4058-e548-4a56-9017-5b57dad353ce-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473616 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473628 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473640 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d42e79-429b-48a0-b53f-42a836067d2c-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.473654 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d42e79-429b-48a0-b53f-42a836067d2c-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.474719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.474554 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-podinfo" pod "2f6e4058-e548-4a56-9017-5b57dad353ce" (UID: "2f6e4058-e548-4a56-9017-5b57dad353ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:38:27.475288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.475023 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-kube-api-access-rcqdf" pod "2f6e4058-e548-4a56-9017-5b57dad353ce" (UID: "2f6e4058-e548-4a56-9017-5b57dad353ce"). InnerVolumeSpecName "kube-api-access-rcqdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:27.475288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.475052 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-token" pod "2f6e4058-e548-4a56-9017-5b57dad353ce" (UID: "2f6e4058-e548-4a56-9017-5b57dad353ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:27.475944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.475921 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-envoy" pod "2f6e4058-e548-4a56-9017-5b57dad353ce" (UID: "2f6e4058-e548-4a56-9017-5b57dad353ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:27.574381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.574350 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.574381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.574378 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcqdf\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-kube-api-access-rcqdf\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.574381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.574388 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.574664 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.574397 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f6e4058-e548-4a56-9017-5b57dad353ce-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:27.583428 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.583396 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd"] Oct 31 00:38:27.584301 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:38:27.584256 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6bf62d_4d3b_4e98_b70f_0a211a7b3f0e.slice/crio-3c26df5d5f33208577c1ac483fdbe7e5a53d8e81de1a0d49e67ad4ee74f3b90f WatchSource:0}: Error finding container 3c26df5d5f33208577c1ac483fdbe7e5a53d8e81de1a0d49e67ad4ee74f3b90f: Status 404 returned error can't find the container with id 3c26df5d5f33208577c1ac483fdbe7e5a53d8e81de1a0d49e67ad4ee74f3b90f Oct 31 00:38:27.994795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.994685 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" event={"ID":"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e","Type":"ContainerStarted","Data":"313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354"} Oct 31 00:38:27.994795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.994756 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" event={"ID":"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e","Type":"ContainerStarted","Data":"3c26df5d5f33208577c1ac483fdbe7e5a53d8e81de1a0d49e67ad4ee74f3b90f"} Oct 31 00:38:27.995346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.994798 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:27.997244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.997217 4969 generic.go:356] "Generic (PLEG): container finished" podID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerID="b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4" exitCode=0 Oct 31 00:38:27.997382 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.997357 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" Oct 31 00:38:27.997675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.997652 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" event={"ID":"2f6e4058-e548-4a56-9017-5b57dad353ce","Type":"ContainerDied","Data":"b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4"} Oct 31 00:38:27.997779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.997713 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr" event={"ID":"2f6e4058-e548-4a56-9017-5b57dad353ce","Type":"ContainerDied","Data":"79460c2d57d922453b68791cf10e6942f4ef76e77bcd88c58ad199b9d5aa050b"} Oct 31 00:38:27.997779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.997745 4969 scope.go:122] "RemoveContainer" containerID="b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4" Oct 31 00:38:27.999832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:27.999780 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" Oct 31 00:38:28.000532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.000490 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" event={"ID":"61d42e79-429b-48a0-b53f-42a836067d2c","Type":"ContainerDied","Data":"2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24"} Oct 31 00:38:28.000908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.000863 4969 generic.go:356] "Generic (PLEG): container finished" podID="61d42e79-429b-48a0-b53f-42a836067d2c" containerID="2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24" exitCode=0 Oct 31 00:38:28.001037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.000952 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm" event={"ID":"61d42e79-429b-48a0-b53f-42a836067d2c","Type":"ContainerDied","Data":"d56da74098cff113e609bf80639d59f6023a97f97ad6b14f9db6b195af4dd720"} Oct 31 00:38:28.013982 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.013958 4969 scope.go:122] "RemoveContainer" containerID="f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc" Oct 31 00:38:28.024036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.024014 4969 scope.go:122] "RemoveContainer" containerID="b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4" Oct 31 00:38:28.029138 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:38:28.029114 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4\": container with ID starting with b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4 not found: ID does not exist" containerID="b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4" Oct 31 00:38:28.029231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.029146 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4"} err="failed to get container status \"b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4\": rpc error: code = NotFound desc = could not find container \"b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4\": container with ID starting with b873ea2748e2380cd4551941f42a1535090a6f4608c37a4276c5e923d2914aa4 not found: ID does not exist" Oct 31 00:38:28.029231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.029164 4969 scope.go:122] "RemoveContainer" containerID="f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc" Oct 31 00:38:28.029725 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:38:28.029700 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc\": container with ID starting with f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc not found: ID does not exist" containerID="f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc" Oct 31 00:38:28.029822 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.029731 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc"} err="failed to get container status \"f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc\": rpc error: code = NotFound desc = could not find container \"f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc\": container with ID starting with f41e21fe971834a81da2ebabceac22965c5f4e89dd66c77c38136ee3295bdfcc not found: ID does not exist" Oct 31 00:38:28.029822 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.029748 4969 scope.go:122] "RemoveContainer" containerID="2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24" Oct 31 00:38:28.053740 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.053722 4969 scope.go:122] "RemoveContainer" containerID="2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24" Oct 31 00:38:28.054029 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:38:28.054007 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24\": container with ID starting with 2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24 not found: ID does not exist" containerID="2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24" Oct 31 00:38:28.054081 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.054040 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24"} err="failed to get container status \"2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24\": rpc error: code = NotFound desc = could not find container \"2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24\": container with ID starting with 2bb81c1b5fd1c14d474f9820611a8327c3a4572f2dae1623672850bb85739c24 not found: ID does not exist" Oct 31 00:38:28.110097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.110047 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" podStartSLOduration=2.110034455 podStartE2EDuration="2.110034455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:28.10996011 +0000 UTC m=+2140.585318533" watchObservedRunningTime="2026-10-31 00:38:28.110034455 +0000 UTC m=+2140.585392936" Oct 31 00:38:28.229339 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.229293 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr"] Oct 31 00:38:28.232844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.232818 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-thkzr"] Oct 31 00:38:28.349036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.348903 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm"] Oct 31 00:38:28.352703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:28.352680 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-6bnvm"] Oct 31 00:38:29.735146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:29.735113 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" path="/var/lib/kubelet/pods/2f6e4058-e548-4a56-9017-5b57dad353ce/volumes" Oct 31 00:38:29.735629 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:29.735585 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" path="/var/lib/kubelet/pods/61d42e79-429b-48a0-b53f-42a836067d2c/volumes" Oct 31 00:38:32.735634 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:38:32.735577 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:38:36.733311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:36.733275 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9"] Oct 31 00:38:37.671983 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.671950 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd"] Oct 31 00:38:37.672577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.672564 4969 cpu_manager.go:414] "containerMap: removing container" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" Oct 31 00:38:37.672642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.672579 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" Oct 31 00:38:37.672642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.672619 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" Oct 31 00:38:37.672642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.672625 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" Oct 31 00:38:37.672754 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.672745 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" Oct 31 00:38:37.672795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.672760 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d42e79-429b-48a0-b53f-42a836067d2c" containerName="istio-proxy" Oct 31 00:38:37.672795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.672768 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" Oct 31 00:38:37.677552 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.677532 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.729959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.729929 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2c02a9-openshift-default-dockercfg-lscnf\"" Oct 31 00:38:37.741360 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.741327 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd"] Oct 31 00:38:37.817422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.817386 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-certs\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.817422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.817428 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-envoy\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.817655 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.817536 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcwgb\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-kube-api-access-dcwgb\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.817655 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.817636 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-data\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.817773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.817753 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istiod-ca-cert\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.817954 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.817809 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-socket\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.818166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.818073 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-token\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.818166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.818133 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-credential-socket\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.818428 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.818210 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-podinfo\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.919776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.919738 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-data\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.919942 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.919810 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istiod-ca-cert\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.919942 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.919846 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-socket\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.919942 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.919873 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-token\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.920228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.920086 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-credential-socket\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.921343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.920321 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-socket\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.921343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.920368 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-data\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.921343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.920127 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-podinfo\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.921343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.920709 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-credential-socket\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.921343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.921034 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-certs\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.921343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.921094 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-envoy\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.921343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.921199 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istiod-ca-cert\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.921343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.921273 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcwgb\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-kube-api-access-dcwgb\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.921343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.921314 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-certs\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.923324 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.923273 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-podinfo\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.923669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.923650 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-envoy\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.984481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.984446 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-token\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.986816 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.986783 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcwgb\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-kube-api-access-dcwgb\") pod \"e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:37.991549 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:37.991528 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:38.134483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.134447 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb"] Oct 31 00:38:38.135513 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.135492 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" Oct 31 00:38:38.135564 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.135519 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6e4058-e548-4a56-9017-5b57dad353ce" containerName="istio-proxy" Oct 31 00:38:38.141179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.141156 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:38.193461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.193380 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f6a8a5\"" Oct 31 00:38:38.193694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.193668 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f6a8a5-dockercfg-pxb5r\"" Oct 31 00:38:38.194730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.194705 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb"] Oct 31 00:38:38.231635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.229225 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrc4f\" (UniqueName: \"kubernetes.io/projected/aab82b8a-cef9-49f2-a946-81604ca84724-kube-api-access-nrc4f\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb\" (UID: \"aab82b8a-cef9-49f2-a946-81604ca84724\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:38.231635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.229520 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab82b8a-cef9-49f2-a946-81604ca84724-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb\" (UID: \"aab82b8a-cef9-49f2-a946-81604ca84724\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:38.331477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.331439 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab82b8a-cef9-49f2-a946-81604ca84724-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb\" (UID: \"aab82b8a-cef9-49f2-a946-81604ca84724\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:38.331772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.331545 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrc4f\" (UniqueName: \"kubernetes.io/projected/aab82b8a-cef9-49f2-a946-81604ca84724-kube-api-access-nrc4f\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb\" (UID: \"aab82b8a-cef9-49f2-a946-81604ca84724\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:38.332216 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.332186 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab82b8a-cef9-49f2-a946-81604ca84724-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb\" (UID: \"aab82b8a-cef9-49f2-a946-81604ca84724\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:38.394191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.394153 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrc4f\" (UniqueName: \"kubernetes.io/projected/aab82b8a-cef9-49f2-a946-81604ca84724-kube-api-access-nrc4f\") pod \"payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb\" (UID: \"aab82b8a-cef9-49f2-a946-81604ca84724\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:38.453552 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:38:38.453515 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30e3b7f_f8d0_4d37_966a_0d3f6e0820b0.slice/crio-74b96a984e2cfbdce52385549f4d310a7d1a53ee00b1ad74c0b19c05bbcdb9b4 WatchSource:0}: Error finding container 74b96a984e2cfbdce52385549f4d310a7d1a53ee00b1ad74c0b19c05bbcdb9b4: Status 404 returned error can't find the container with id 74b96a984e2cfbdce52385549f4d310a7d1a53ee00b1ad74c0b19c05bbcdb9b4 Oct 31 00:38:38.454798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.454759 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd"] Oct 31 00:38:38.457060 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.457017 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:38.457195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.457109 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:38.457195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.457154 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:38.459781 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.459756 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:38.848052 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.848024 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb"] Oct 31 00:38:38.848764 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:38:38.848734 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab82b8a_cef9_49f2_a946_81604ca84724.slice/crio-da2c74d05ef0e1dd28b796f3aa5724694479739bf1cb06015979380ba5e0361d WatchSource:0}: Error finding container da2c74d05ef0e1dd28b796f3aa5724694479739bf1cb06015979380ba5e0361d: Status 404 returned error can't find the container with id da2c74d05ef0e1dd28b796f3aa5724694479739bf1cb06015979380ba5e0361d Oct 31 00:38:38.927146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.927113 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw"] Oct 31 00:38:38.932351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.932325 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:38.987844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:38.987811 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw"] Oct 31 00:38:39.039005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.038970 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnzc\" (UniqueName: \"kubernetes.io/projected/c11b9467-7924-4ac3-bdc7-f05586998b88-kube-api-access-jjnzc\") pod \"payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw\" (UID: \"c11b9467-7924-4ac3-bdc7-f05586998b88\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:39.039174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.039028 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11b9467-7924-4ac3-bdc7-f05586998b88-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw\" (UID: \"c11b9467-7924-4ac3-bdc7-f05586998b88\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:39.054195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.054168 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:38:39.139911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.139876 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnzc\" (UniqueName: \"kubernetes.io/projected/c11b9467-7924-4ac3-bdc7-f05586998b88-kube-api-access-jjnzc\") pod \"payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw\" (UID: \"c11b9467-7924-4ac3-bdc7-f05586998b88\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:39.139911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.139891 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" event={"ID":"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0","Type":"ContainerStarted","Data":"cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604"} Oct 31 00:38:39.140155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.139942 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" event={"ID":"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0","Type":"ContainerStarted","Data":"74b96a984e2cfbdce52385549f4d310a7d1a53ee00b1ad74c0b19c05bbcdb9b4"} Oct 31 00:38:39.140155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.139946 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11b9467-7924-4ac3-bdc7-f05586998b88-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw\" (UID: \"c11b9467-7924-4ac3-bdc7-f05586998b88\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:39.140762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.140742 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11b9467-7924-4ac3-bdc7-f05586998b88-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw\" (UID: \"c11b9467-7924-4ac3-bdc7-f05586998b88\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:39.141675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.141648 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" event={"ID":"aab82b8a-cef9-49f2-a946-81604ca84724","Type":"ContainerStarted","Data":"44953ffe10b8e18d33a0f0ec10f092cc8111be23c038cfcc825cbd50c4a8494d"} Oct 31 00:38:39.141751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.141705 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:39.141751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.141731 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" event={"ID":"aab82b8a-cef9-49f2-a946-81604ca84724","Type":"ContainerStarted","Data":"da2c74d05ef0e1dd28b796f3aa5724694479739bf1cb06015979380ba5e0361d"} Oct 31 00:38:39.201325 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.201297 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnzc\" (UniqueName: \"kubernetes.io/projected/c11b9467-7924-4ac3-bdc7-f05586998b88-kube-api-access-jjnzc\") pod \"payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw\" (UID: \"c11b9467-7924-4ac3-bdc7-f05586998b88\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:39.260880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.260854 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:39.295106 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.295053 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" podStartSLOduration=2.295038046 podStartE2EDuration="2.295038046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:39.29087024 +0000 UTC m=+2151.766228662" watchObservedRunningTime="2026-10-31 00:38:39.295038046 +0000 UTC m=+2151.770396468" Oct 31 00:38:39.414091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.412406 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" podStartSLOduration=1.412388627 podStartE2EDuration="1.412388627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:39.410541661 +0000 UTC m=+2151.885900087" watchObservedRunningTime="2026-10-31 00:38:39.412388627 +0000 UTC m=+2151.887747051" Oct 31 00:38:39.649700 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.649668 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw"] Oct 31 00:38:39.650551 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:38:39.650522 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11b9467_7924_4ac3_bdc7_f05586998b88.slice/crio-1f80582c59f9649259d3cf87551c7b1a9d5f9bbed7709769be4e70d42e476973 WatchSource:0}: Error finding container 1f80582c59f9649259d3cf87551c7b1a9d5f9bbed7709769be4e70d42e476973: Status 404 returned error can't find the container with id 1f80582c59f9649259d3cf87551c7b1a9d5f9bbed7709769be4e70d42e476973 Oct 31 00:38:39.992108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.992026 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:39.998029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:39.998004 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:40.151871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:40.151833 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" event={"ID":"c11b9467-7924-4ac3-bdc7-f05586998b88","Type":"ContainerStarted","Data":"223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e"} Oct 31 00:38:40.152057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:40.151894 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" event={"ID":"c11b9467-7924-4ac3-bdc7-f05586998b88","Type":"ContainerStarted","Data":"1f80582c59f9649259d3cf87551c7b1a9d5f9bbed7709769be4e70d42e476973"} Oct 31 00:38:40.152057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:40.151934 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:40.152329 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:40.152285 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:40.153238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:40.153216 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:38:40.269168 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:40.269076 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" podStartSLOduration=2.269061919 podStartE2EDuration="2.269061919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:40.267119312 +0000 UTC m=+2152.742477734" watchObservedRunningTime="2026-10-31 00:38:40.269061919 +0000 UTC m=+2152.744420370" Oct 31 00:38:41.734553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:41.734500 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" podUID="e6a6bf7b-bf37-488d-8ffe-1f2655fd7457" containerName="payload-processing" containerID="cri-o://0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b" gracePeriod=25 Oct 31 00:38:42.013811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.013767 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:38:42.172471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.172436 4969 generic.go:356] "Generic (PLEG): container finished" podID="e6a6bf7b-bf37-488d-8ffe-1f2655fd7457" containerID="0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b" exitCode=0 Oct 31 00:38:42.172709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.172494 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" Oct 31 00:38:42.172709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.172521 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" event={"ID":"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457","Type":"ContainerDied","Data":"0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b"} Oct 31 00:38:42.172709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.172577 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9" event={"ID":"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457","Type":"ContainerDied","Data":"d158ff6f9784b0956f8e8034264aa8d3dae23ce4abf81e080299680dc67cf18e"} Oct 31 00:38:42.172709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.172624 4969 scope.go:122] "RemoveContainer" containerID="0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b" Oct 31 00:38:42.181447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.181418 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-plugins-config-volume\") pod \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\" (UID: \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\") " Oct 31 00:38:42.181611 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.181565 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-kube-api-access-6n6nj\" (UniqueName: \"kubernetes.io/projected/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-kube-api-access-6n6nj\") pod \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\" (UID: \"e6a6bf7b-bf37-488d-8ffe-1f2655fd7457\") " Oct 31 00:38:42.181860 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.181839 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-plugins-config-volume" pod "e6a6bf7b-bf37-488d-8ffe-1f2655fd7457" (UID: "e6a6bf7b-bf37-488d-8ffe-1f2655fd7457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:42.182097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.182079 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.184232 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.184210 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-kube-api-access-6n6nj" pod "e6a6bf7b-bf37-488d-8ffe-1f2655fd7457" (UID: "e6a6bf7b-bf37-488d-8ffe-1f2655fd7457"). InnerVolumeSpecName "kube-api-access-6n6nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:42.184333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.184242 4969 scope.go:122] "RemoveContainer" containerID="0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b" Oct 31 00:38:42.184646 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:38:42.184578 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b\": container with ID starting with 0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b not found: ID does not exist" containerID="0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b" Oct 31 00:38:42.184712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.184655 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b"} err="failed to get container status \"0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b\": rpc error: code = NotFound desc = could not find container \"0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b\": container with ID starting with 0bea4d4487d748e1df02793df8e8a5ab4d888f96f58e4386b5def79f0266a18b not found: ID does not exist" Oct 31 00:38:42.283358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.283265 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n6nj\" (UniqueName: \"kubernetes.io/projected/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457-kube-api-access-6n6nj\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:38:42.598241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.598155 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9"] Oct 31 00:38:42.605337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:42.605313 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-687d7d4d75-fp6f9"] Oct 31 00:38:43.735648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:43.735616 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a6bf7b-bf37-488d-8ffe-1f2655fd7457" path="/var/lib/kubelet/pods/e6a6bf7b-bf37-488d-8ffe-1f2655fd7457/volumes" Oct 31 00:38:47.734533 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.734503 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b"] Oct 31 00:38:47.735077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.735062 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e6a6bf7b-bf37-488d-8ffe-1f2655fd7457" containerName="payload-processing" Oct 31 00:38:47.735119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.735078 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a6bf7b-bf37-488d-8ffe-1f2655fd7457" containerName="payload-processing" Oct 31 00:38:47.735205 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.735195 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a6bf7b-bf37-488d-8ffe-1f2655fd7457" containerName="payload-processing" Oct 31 00:38:47.739804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.739785 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.797581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.797554 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-87b0f8-openshift-default-dockercfg-whqlc\"" Oct 31 00:38:47.848800 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.848767 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b"] Oct 31 00:38:47.850681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.850656 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-envoy\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.850879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.850709 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-podinfo\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.850879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.850831 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-credential-socket\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.850879 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.850863 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-certs\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.851097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.850923 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-data\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.851097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.850987 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-socket\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.851097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.851049 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-token\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.851097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.851089 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn22m\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-kube-api-access-mn22m\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.851271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.851134 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f0772e-b63a-4552-8483-98f88c7a3fd7-istiod-ca-cert\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.952645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.952579 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn22m\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-kube-api-access-mn22m\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.952847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.952695 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f0772e-b63a-4552-8483-98f88c7a3fd7-istiod-ca-cert\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.952847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.952789 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-envoy\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.952992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.952889 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-podinfo\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.952992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.952932 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-credential-socket\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.952992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.952965 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-certs\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.953139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.953027 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-data\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.953139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.953098 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-socket\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.953263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.953196 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-token\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.953342 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.953306 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-certs\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.953632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.953562 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f0772e-b63a-4552-8483-98f88c7a3fd7-istiod-ca-cert\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.953632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.953585 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-socket\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.953806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.953639 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-data\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.953847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.953813 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-credential-socket\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.955278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.955258 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-envoy\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:47.955546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:47.955528 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-podinfo\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:48.013732 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:48.013660 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-token\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:48.013883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:48.013774 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn22m\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-kube-api-access-mn22m\") pod \"e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:48.053726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:48.053697 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:48.489443 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:48.489415 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b"] Oct 31 00:38:48.489917 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:38:48.489895 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f0772e_b63a_4552_8483_98f88c7a3fd7.slice/crio-1195dc36cd97fc3c792b018d8a468e19d3b4cb1a79d96e923e5c2ba07f098619 WatchSource:0}: Error finding container 1195dc36cd97fc3c792b018d8a468e19d3b4cb1a79d96e923e5c2ba07f098619: Status 404 returned error can't find the container with id 1195dc36cd97fc3c792b018d8a468e19d3b4cb1a79d96e923e5c2ba07f098619 Oct 31 00:38:48.492404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:48.492368 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:48.492480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:48.492435 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:48.492480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:48.492469 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:49.145960 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:49.145927 4969 scope.go:122] "RemoveContainer" containerID="5c7c33b4e77d86edc4213e09e923f5609e8d1140890ba5b06c55f7694048e7e0" Oct 31 00:38:49.159235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:49.159210 4969 scope.go:122] "RemoveContainer" containerID="e3f0aff8db4e601eb45f0ea4dd75509d9da7431af3848171dcc28834be9f7b09" Oct 31 00:38:49.233008 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:49.232977 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" event={"ID":"16f0772e-b63a-4552-8483-98f88c7a3fd7","Type":"ContainerStarted","Data":"8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790"} Oct 31 00:38:49.233160 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:49.233027 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" event={"ID":"16f0772e-b63a-4552-8483-98f88c7a3fd7","Type":"ContainerStarted","Data":"1195dc36cd97fc3c792b018d8a468e19d3b4cb1a79d96e923e5c2ba07f098619"} Oct 31 00:38:49.353503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:49.353280 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" podStartSLOduration=2.353260421 podStartE2EDuration="2.353260421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:49.352570527 +0000 UTC m=+2161.827928951" watchObservedRunningTime="2026-10-31 00:38:49.353260421 +0000 UTC m=+2161.828618847" Oct 31 00:38:50.054716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:50.054686 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:50.060452 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:50.060425 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:50.154403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:50.154370 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:38:50.241434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:50.241404 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:50.242474 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:50.242452 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:38:51.163227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:51.163202 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:38:56.064619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.060636 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc"] Oct 31 00:38:56.078977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.078948 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:38:56.121573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.121536 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc"] Oct 31 00:38:56.166716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.166682 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-87b0f8\"" Oct 31 00:38:56.166900 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.166731 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-87b0f8-dockercfg-vvpcl\"" Oct 31 00:38:56.173363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.173332 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfsbz\" (UniqueName: \"kubernetes.io/projected/f33ee314-f517-410c-8dff-bc0ebe2ae914-kube-api-access-bfsbz\") pod \"payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc\" (UID: \"f33ee314-f517-410c-8dff-bc0ebe2ae914\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:38:56.173509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.173486 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ee314-f517-410c-8dff-bc0ebe2ae914-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc\" (UID: \"f33ee314-f517-410c-8dff-bc0ebe2ae914\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:38:56.274980 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.274945 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfsbz\" (UniqueName: \"kubernetes.io/projected/f33ee314-f517-410c-8dff-bc0ebe2ae914-kube-api-access-bfsbz\") pod \"payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc\" (UID: \"f33ee314-f517-410c-8dff-bc0ebe2ae914\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:38:56.275149 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.275053 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ee314-f517-410c-8dff-bc0ebe2ae914-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc\" (UID: \"f33ee314-f517-410c-8dff-bc0ebe2ae914\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:38:56.276019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.275997 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ee314-f517-410c-8dff-bc0ebe2ae914-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc\" (UID: \"f33ee314-f517-410c-8dff-bc0ebe2ae914\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:38:56.335965 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.335899 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfsbz\" (UniqueName: \"kubernetes.io/projected/f33ee314-f517-410c-8dff-bc0ebe2ae914-kube-api-access-bfsbz\") pod \"payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc\" (UID: \"f33ee314-f517-410c-8dff-bc0ebe2ae914\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:38:56.398090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.398058 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:38:56.781194 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:38:56.781157 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33ee314_f517_410c_8dff_bc0ebe2ae914.slice/crio-b1a7abc489ee5cecda80ade64c23db0ef7bd1191e05fa50d1db7b2fc5da49c1d WatchSource:0}: Error finding container b1a7abc489ee5cecda80ade64c23db0ef7bd1191e05fa50d1db7b2fc5da49c1d: Status 404 returned error can't find the container with id b1a7abc489ee5cecda80ade64c23db0ef7bd1191e05fa50d1db7b2fc5da49c1d Oct 31 00:38:56.781330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:56.781304 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc"] Oct 31 00:38:57.297679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:57.297637 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" event={"ID":"f33ee314-f517-410c-8dff-bc0ebe2ae914","Type":"ContainerStarted","Data":"5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa"} Oct 31 00:38:57.298061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:57.297695 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" event={"ID":"f33ee314-f517-410c-8dff-bc0ebe2ae914","Type":"ContainerStarted","Data":"b1a7abc489ee5cecda80ade64c23db0ef7bd1191e05fa50d1db7b2fc5da49c1d"} Oct 31 00:38:57.298061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:57.297733 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:38:57.415080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:38:57.415028 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" podStartSLOduration=1.41501366 podStartE2EDuration="1.41501366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:57.412810581 +0000 UTC m=+2169.888169006" watchObservedRunningTime="2026-10-31 00:38:57.41501366 +0000 UTC m=+2169.890372083" Oct 31 00:39:02.729880 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:39:02.729843 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:39:07.712234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:07.712193 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t_b115e833-ff1c-4d0a-be5f-13126f600f21/payload-processing/0.log" Oct 31 00:39:08.307937 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:08.307906 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:39:08.685624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:08.685583 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl_70ecefa7-43bc-4a70-950e-2e28d2dc6e43/payload-processing/0.log" Oct 31 00:39:08.926509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:08.926483 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl_70ecefa7-43bc-4a70-950e-2e28d2dc6e43/payload-processing/0.log" Oct 31 00:39:24.314462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.314428 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm"] Oct 31 00:39:24.318705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.318685 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.372113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.372084 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-726d5b-serving-cert\"" Oct 31 00:39:24.426518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.426484 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm"] Oct 31 00:39:24.429765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.429742 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-metrics-certs\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.429847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.429787 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdceb92-a871-45c3-9729-306e0dceb327-openshift-service-ca\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.429912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.429900 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldw4\" (UniqueName: \"kubernetes.io/projected/bcdceb92-a871-45c3-9729-306e0dceb327-kube-api-access-nldw4\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.430017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.430004 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.531088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.531055 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldw4\" (UniqueName: \"kubernetes.io/projected/bcdceb92-a871-45c3-9729-306e0dceb327-kube-api-access-nldw4\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.531250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.531159 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.531250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.531225 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-metrics-certs\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.531337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.531254 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdceb92-a871-45c3-9729-306e0dceb327-openshift-service-ca\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.531337 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:39:24.531297 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-serving-cert: secret "maas-api-e2e-api-b-726d5b-serving-cert" not found Oct 31 00:39:24.531429 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:39:24.531372 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls podName:bcdceb92-a871-45c3-9729-306e0dceb327 nodeName:}" failed. No retries permitted until 2026-10-31 00:39:25.031351649 +0000 UTC m=+2197.506710054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls") pod "maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" (UID: "bcdceb92-a871-45c3-9729-306e0dceb327") : secret "maas-api-e2e-api-b-726d5b-serving-cert" not found Oct 31 00:39:24.531914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.531892 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdceb92-a871-45c3-9729-306e0dceb327-openshift-service-ca\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.534023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.533998 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-metrics-certs\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:24.592827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:24.592758 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldw4\" (UniqueName: \"kubernetes.io/projected/bcdceb92-a871-45c3-9729-306e0dceb327-kube-api-access-nldw4\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:25.038367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:25.038329 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:25.041138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:25.041104 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls\") pod \"maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:25.232267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:25.232235 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:25.616568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:25.616526 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm"] Oct 31 00:39:25.617169 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:39:25.617136 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdceb92_a871_45c3_9729_306e0dceb327.slice/crio-e10f166f0af91cf96e687f70a65a14b447d064c33d982cdc0e9fafd9fd377a9c WatchSource:0}: Error finding container e10f166f0af91cf96e687f70a65a14b447d064c33d982cdc0e9fafd9fd377a9c: Status 404 returned error can't find the container with id e10f166f0af91cf96e687f70a65a14b447d064c33d982cdc0e9fafd9fd377a9c Oct 31 00:39:26.510790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:26.510755 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" event={"ID":"bcdceb92-a871-45c3-9729-306e0dceb327","Type":"ContainerStarted","Data":"e10f166f0af91cf96e687f70a65a14b447d064c33d982cdc0e9fafd9fd377a9c"} Oct 31 00:39:27.519395 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:27.519363 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" event={"ID":"bcdceb92-a871-45c3-9729-306e0dceb327","Type":"ContainerStarted","Data":"f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef"} Oct 31 00:39:27.519897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:27.519568 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:27.637105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:27.637041 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" podStartSLOduration=2.5658370809999997 podStartE2EDuration="3.637022465s" totalImagesPullingTime="1.071185384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:27.636505834 +0000 UTC m=+2200.111864256" watchObservedRunningTime="2026-10-31 00:39:27.637022465 +0000 UTC m=+2200.112380894" Oct 31 00:39:32.728869 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:39:32.728821 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:39:33.532561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:33.532528 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:39:39.666618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.665804 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6"] Oct 31 00:39:39.672142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.672120 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:39.720330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.720299 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6"] Oct 31 00:39:39.724159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.724121 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg6tk\" (UniqueName: \"kubernetes.io/projected/74ff306f-d736-452d-831a-b0a3aa39b3ee-kube-api-access-lg6tk\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6\" (UID: \"74ff306f-d736-452d-831a-b0a3aa39b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:39.724303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.724180 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ff306f-d736-452d-831a-b0a3aa39b3ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6\" (UID: \"74ff306f-d736-452d-831a-b0a3aa39b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:39.764413 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.764376 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-97fc39\"" Oct 31 00:39:39.764558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.764449 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-97fc39-dockercfg-f8nvj\"" Oct 31 00:39:39.826124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.826086 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg6tk\" (UniqueName: \"kubernetes.io/projected/74ff306f-d736-452d-831a-b0a3aa39b3ee-kube-api-access-lg6tk\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6\" (UID: \"74ff306f-d736-452d-831a-b0a3aa39b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:39.826297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.826135 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ff306f-d736-452d-831a-b0a3aa39b3ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6\" (UID: \"74ff306f-d736-452d-831a-b0a3aa39b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:39.827057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.827014 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ff306f-d736-452d-831a-b0a3aa39b3ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6\" (UID: \"74ff306f-d736-452d-831a-b0a3aa39b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:39.887274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:39.887243 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg6tk\" (UniqueName: \"kubernetes.io/projected/74ff306f-d736-452d-831a-b0a3aa39b3ee-kube-api-access-lg6tk\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6\" (UID: \"74ff306f-d736-452d-831a-b0a3aa39b3ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:40.001514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:40.001435 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:40.380563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:40.380532 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6"] Oct 31 00:39:40.380765 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:39:40.380742 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ff306f_d736_452d_831a_b0a3aa39b3ee.slice/crio-75ede7f7feee8e01fefee1311de26906006468c6b010681c56b0a8c648e973c1 WatchSource:0}: Error finding container 75ede7f7feee8e01fefee1311de26906006468c6b010681c56b0a8c648e973c1: Status 404 returned error can't find the container with id 75ede7f7feee8e01fefee1311de26906006468c6b010681c56b0a8c648e973c1 Oct 31 00:39:40.634231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:40.634138 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" event={"ID":"74ff306f-d736-452d-831a-b0a3aa39b3ee","Type":"ContainerStarted","Data":"ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550"} Oct 31 00:39:40.634231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:40.634193 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" event={"ID":"74ff306f-d736-452d-831a-b0a3aa39b3ee","Type":"ContainerStarted","Data":"75ede7f7feee8e01fefee1311de26906006468c6b010681c56b0a8c648e973c1"} Oct 31 00:39:40.634231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:40.634218 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:40.751692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:40.751639 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" podStartSLOduration=1.751624601 podStartE2EDuration="1.751624601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:40.75038265 +0000 UTC m=+2213.225741074" watchObservedRunningTime="2026-10-31 00:39:40.751624601 +0000 UTC m=+2213.226983024" Oct 31 00:39:47.693863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.693831 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp"] Oct 31 00:39:47.698113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.698095 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.752009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.751981 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2c02a9-serving-cert\"" Oct 31 00:39:47.805604 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.805567 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp"] Oct 31 00:39:47.823335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.823282 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.823519 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.823478 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.823872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.823839 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wm6\" (UniqueName: \"kubernetes.io/projected/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-kube-api-access-m8wm6\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.823992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.823897 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-metrics-certs\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.925213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.925179 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.925398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.925320 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wm6\" (UniqueName: \"kubernetes.io/projected/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-kube-api-access-m8wm6\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.925398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.925347 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-metrics-certs\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.925483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.925396 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.925544 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:39:47.925528 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-serving-cert: secret "maas-api-e2e-ipp-b-2c02a9-serving-cert" not found Oct 31 00:39:47.925624 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:39:47.925612 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls podName:271b11b2-1a87-4c63-aa4b-8d2f56b28fc0 nodeName:}" failed. No retries permitted until 2026-10-31 00:39:48.425572087 +0000 UTC m=+2220.900930507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls") pod "maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" (UID: "271b11b2-1a87-4c63-aa4b-8d2f56b28fc0") : secret "maas-api-e2e-ipp-b-2c02a9-serving-cert" not found Oct 31 00:39:47.925963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.925941 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.928148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.928124 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-metrics-certs\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:47.987691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:47.987582 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wm6\" (UniqueName: \"kubernetes.io/projected/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-kube-api-access-m8wm6\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:48.430982 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:48.430946 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:48.433553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:48.433527 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:48.612404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:48.612375 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:48.999653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:48.999620 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp"] Oct 31 00:39:49.000125 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:39:48.999696 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271b11b2_1a87_4c63_aa4b_8d2f56b28fc0.slice/crio-671d0244a6fa27dcd78b271916fb12ec24a4dfecb8117c320d6111afccb6f38a WatchSource:0}: Error finding container 671d0244a6fa27dcd78b271916fb12ec24a4dfecb8117c320d6111afccb6f38a: Status 404 returned error can't find the container with id 671d0244a6fa27dcd78b271916fb12ec24a4dfecb8117c320d6111afccb6f38a Oct 31 00:39:49.212364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:49.212333 4969 scope.go:122] "RemoveContainer" containerID="2d7da94deb0fd2e39da7fc06227652d2acbea750f21cca65af5c9b1c85bf4c4e" Oct 31 00:39:49.711387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:49.711351 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" event={"ID":"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0","Type":"ContainerStarted","Data":"671d0244a6fa27dcd78b271916fb12ec24a4dfecb8117c320d6111afccb6f38a"} Oct 31 00:39:50.739953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:50.739906 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" event={"ID":"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0","Type":"ContainerStarted","Data":"4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68"} Oct 31 00:39:50.741106 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:50.741079 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:39:50.862403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:50.862349 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" podStartSLOduration=3.006611044 podStartE2EDuration="3.86233429s" totalImagesPullingTime="855.723246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:50.857575473 +0000 UTC m=+2223.332933896" watchObservedRunningTime="2026-10-31 00:39:50.86233429 +0000 UTC m=+2223.337692723" Oct 31 00:39:51.645770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:51.645740 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:39:57.760162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:39:57.760134 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:40:02.744331 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:40:02.744289 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:40:13.804389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:13.804356 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr"] Oct 31 00:40:13.808788 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:13.808768 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:13.912417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:13.912383 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1a4f30-serving-cert\"" Oct 31 00:40:13.916691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:13.916664 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr"] Oct 31 00:40:13.960961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:13.960926 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69df93e1-3356-4017-b8fb-6429c2383bab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:13.961187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:13.961168 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:13.961268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:13.961216 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65l4c\" (UniqueName: \"kubernetes.io/projected/69df93e1-3356-4017-b8fb-6429c2383bab-kube-api-access-65l4c\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:13.961351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:13.961313 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.062413 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.062314 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.062413 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.062393 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65l4c\" (UniqueName: \"kubernetes.io/projected/69df93e1-3356-4017-b8fb-6429c2383bab-kube-api-access-65l4c\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.062413 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.062421 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.062701 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.062508 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69df93e1-3356-4017-b8fb-6429c2383bab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.062701 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:40:14.062636 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-serving-cert: secret "maas-api-e2e-ipp-b-1a4f30-serving-cert" not found Oct 31 00:40:14.062806 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:40:14.062745 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls podName:69df93e1-3356-4017-b8fb-6429c2383bab nodeName:}" failed. No retries permitted until 2026-10-31 00:40:14.562721417 +0000 UTC m=+2247.038079909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls") pod "maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" (UID: "69df93e1-3356-4017-b8fb-6429c2383bab") : secret "maas-api-e2e-ipp-b-1a4f30-serving-cert" not found Oct 31 00:40:14.063450 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.063425 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69df93e1-3356-4017-b8fb-6429c2383bab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.067914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.067886 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-metrics-certs\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.124984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.124950 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65l4c\" (UniqueName: \"kubernetes.io/projected/69df93e1-3356-4017-b8fb-6429c2383bab-kube-api-access-65l4c\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.570031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.569993 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.572632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.572608 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:14.727622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:14.727097 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:15.160634 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:40:15.160196 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69df93e1_3356_4017_b8fb_6429c2383bab.slice/crio-801ce049356133d11d5e7b91b41fe7154f4ef67c1fc89ff7e79b775759702dcf WatchSource:0}: Error finding container 801ce049356133d11d5e7b91b41fe7154f4ef67c1fc89ff7e79b775759702dcf: Status 404 returned error can't find the container with id 801ce049356133d11d5e7b91b41fe7154f4ef67c1fc89ff7e79b775759702dcf Oct 31 00:40:15.160634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:15.160608 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr"] Oct 31 00:40:15.949858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:15.949824 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" event={"ID":"69df93e1-3356-4017-b8fb-6429c2383bab","Type":"ContainerStarted","Data":"801ce049356133d11d5e7b91b41fe7154f4ef67c1fc89ff7e79b775759702dcf"} Oct 31 00:40:16.960251 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:16.960219 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" event={"ID":"69df93e1-3356-4017-b8fb-6429c2383bab","Type":"ContainerStarted","Data":"f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c"} Oct 31 00:40:16.961159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:16.961139 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:21.285819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:21.285768 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:40:21.286428 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:21.285830 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:40:24.008123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.008096 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:40:24.051260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.051184 4969 generic.go:356] "Generic (PLEG): container finished" podID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerID="8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790" exitCode=0 Oct 31 00:40:24.051402 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.051257 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" event={"ID":"16f0772e-b63a-4552-8483-98f88c7a3fd7","Type":"ContainerDied","Data":"8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790"} Oct 31 00:40:24.064070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.052230 4969 scope.go:122] "RemoveContainer" containerID="8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790" Oct 31 00:40:24.064070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.058362 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:40:24.064070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.058520 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:40:24.064070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.058641 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:40:24.065163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.064579 4969 generic.go:356] "Generic (PLEG): container finished" podID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerID="cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604" exitCode=0 Oct 31 00:40:24.065163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.064659 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" event={"ID":"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0","Type":"ContainerDied","Data":"cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604"} Oct 31 00:40:24.065322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.065218 4969 scope.go:122] "RemoveContainer" containerID="cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604" Oct 31 00:40:24.075089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.075057 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:40:24.075197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.075127 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:40:24.075197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.075158 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:40:24.129534 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:24.129477 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" podStartSLOduration=10.22277609 podStartE2EDuration="11.129460403s" totalImagesPullingTime="906.684313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:17.094498425 +0000 UTC m=+2249.569856870" watchObservedRunningTime="2026-10-31 00:40:24.129460403 +0000 UTC m=+2256.604818827" Oct 31 00:40:25.074018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:25.073987 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" event={"ID":"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0","Type":"ContainerStarted","Data":"144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4"} Oct 31 00:40:25.076128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:25.076101 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" event={"ID":"16f0772e-b63a-4552-8483-98f88c7a3fd7","Type":"ContainerStarted","Data":"91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710"} Oct 31 00:40:25.152275 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:25.152232 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:40:25.996622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:25.994913 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:40:25.996622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:25.995068 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 00:40:25.996622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:25.995112 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 00:40:26.054583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:26.054549 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:40:26.054795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:26.054766 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 31 00:40:26.054850 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:26.054832 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 31 00:40:27.013621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:26.998978 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 00:40:27.013621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:26.999039 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 00:40:27.054574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:27.054542 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 31 00:40:27.054793 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:27.054616 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 31 00:40:27.997138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:27.997092 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 00:40:27.997494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:27.997462 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 00:40:28.054525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:28.054486 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 31 00:40:28.055049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:28.054834 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 31 00:40:28.998011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:28.997969 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 00:40:28.998297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:28.998266 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 00:40:29.054606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:29.054554 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 31 00:40:29.054992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:29.054634 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 31 00:40:30.008620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:30.007710 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 31 00:40:30.008620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:30.007775 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 31 00:40:30.055030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:30.054993 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 31 00:40:30.055435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:30.055048 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 31 00:40:30.996726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:30.996694 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:40:30.997911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:30.997888 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:40:31.058226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:31.058201 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:40:31.058607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:31.058471 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:40:31.059197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:31.059177 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:40:32.729183 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:40:32.729144 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:40:58.740302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:40:58.740223 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt"] Oct 31 00:41:02.728522 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:41:02.728485 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:41:03.740818 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:03.740777 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" podUID="66768b27-3dd7-4b05-a1e7-e936dccb7e7c" containerName="payload-pre-processing" containerID="cri-o://44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e" gracePeriod=25 Oct 31 00:41:04.030381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.030355 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:41:04.146989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.146955 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-plugins-config-volume\") pod \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\" (UID: \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\") " Oct 31 00:41:04.147187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.147067 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-kube-api-access-4rtmt\" (UniqueName: \"kubernetes.io/projected/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-kube-api-access-4rtmt\") pod \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\" (UID: \"66768b27-3dd7-4b05-a1e7-e936dccb7e7c\") " Oct 31 00:41:04.147392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.147368 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-plugins-config-volume" pod "66768b27-3dd7-4b05-a1e7-e936dccb7e7c" (UID: "66768b27-3dd7-4b05-a1e7-e936dccb7e7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:04.147670 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.147648 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:04.154915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.154878 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-kube-api-access-4rtmt" pod "66768b27-3dd7-4b05-a1e7-e936dccb7e7c" (UID: "66768b27-3dd7-4b05-a1e7-e936dccb7e7c"). InnerVolumeSpecName "kube-api-access-4rtmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:04.249690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.249579 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rtmt\" (UniqueName: \"kubernetes.io/projected/66768b27-3dd7-4b05-a1e7-e936dccb7e7c-kube-api-access-4rtmt\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:04.423173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.423141 4969 generic.go:356] "Generic (PLEG): container finished" podID="66768b27-3dd7-4b05-a1e7-e936dccb7e7c" containerID="44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e" exitCode=0 Oct 31 00:41:04.423349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.423204 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" Oct 31 00:41:04.423349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.423224 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" event={"ID":"66768b27-3dd7-4b05-a1e7-e936dccb7e7c","Type":"ContainerDied","Data":"44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e"} Oct 31 00:41:04.423349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.423313 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt" event={"ID":"66768b27-3dd7-4b05-a1e7-e936dccb7e7c","Type":"ContainerDied","Data":"3b94a50cb4888bbd8eef27a9181fbef15f0a35bdd9bc33267e3b65f5f3a0c00b"} Oct 31 00:41:04.423349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.423341 4969 scope.go:122] "RemoveContainer" containerID="44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e" Oct 31 00:41:04.435715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.435693 4969 scope.go:122] "RemoveContainer" containerID="44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e" Oct 31 00:41:04.436010 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:41:04.435980 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e\": container with ID starting with 44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e not found: ID does not exist" containerID="44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e" Oct 31 00:41:04.436062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.436023 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e"} err="failed to get container status \"44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e\": rpc error: code = NotFound desc = could not find container \"44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e\": container with ID starting with 44485efdb0f81842552c1855a37e1f96c44a4e8806e1ff1bed2a041f0d011b6e not found: ID does not exist" Oct 31 00:41:04.596969 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.596884 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt"] Oct 31 00:41:04.602913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:04.602888 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-752be8-856789d5cf-bhfnt"] Oct 31 00:41:05.735387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:05.735352 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66768b27-3dd7-4b05-a1e7-e936dccb7e7c" path="/var/lib/kubelet/pods/66768b27-3dd7-4b05-a1e7-e936dccb7e7c/volumes" Oct 31 00:41:07.602990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:07.602960 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t"] Oct 31 00:41:07.603381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:07.603148 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" podUID="b115e833-ff1c-4d0a-be5f-13126f600f21" containerName="payload-processing" containerID="cri-o://b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b" gracePeriod=30 Oct 31 00:41:07.718639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:07.710906 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l"] Oct 31 00:41:07.718639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:07.711150 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" podUID="ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8" containerName="payload-pre-processing" containerID="cri-o://faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21" gracePeriod=30 Oct 31 00:41:07.915475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:07.915447 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:41:07.994348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:07.994321 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b115e833-ff1c-4d0a-be5f-13126f600f21-kube-api-access-zgzkx\" (UniqueName: \"kubernetes.io/projected/b115e833-ff1c-4d0a-be5f-13126f600f21-kube-api-access-zgzkx\") pod \"b115e833-ff1c-4d0a-be5f-13126f600f21\" (UID: \"b115e833-ff1c-4d0a-be5f-13126f600f21\") " Oct 31 00:41:07.994473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:07.994455 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b115e833-ff1c-4d0a-be5f-13126f600f21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b115e833-ff1c-4d0a-be5f-13126f600f21-plugins-config-volume\") pod \"b115e833-ff1c-4d0a-be5f-13126f600f21\" (UID: \"b115e833-ff1c-4d0a-be5f-13126f600f21\") " Oct 31 00:41:07.994909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:07.994883 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b115e833-ff1c-4d0a-be5f-13126f600f21-plugins-config-volume" pod "b115e833-ff1c-4d0a-be5f-13126f600f21" (UID: "b115e833-ff1c-4d0a-be5f-13126f600f21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:07.996642 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:07.996583 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b115e833-ff1c-4d0a-be5f-13126f600f21-kube-api-access-zgzkx" pod "b115e833-ff1c-4d0a-be5f-13126f600f21" (UID: "b115e833-ff1c-4d0a-be5f-13126f600f21"). InnerVolumeSpecName "kube-api-access-zgzkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:08.001680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.001659 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:41:08.095377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.095349 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-plugins-config-volume\") pod \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\" (UID: \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\") " Oct 31 00:41:08.095576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.095442 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-kube-api-access-hrkj8\" (UniqueName: \"kubernetes.io/projected/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-kube-api-access-hrkj8\") pod \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\" (UID: \"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8\") " Oct 31 00:41:08.095748 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.095723 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-plugins-config-volume" pod "ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8" (UID: "ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:08.095895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.095880 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b115e833-ff1c-4d0a-be5f-13126f600f21-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:08.095947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.095898 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:08.095947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.095908 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgzkx\" (UniqueName: \"kubernetes.io/projected/b115e833-ff1c-4d0a-be5f-13126f600f21-kube-api-access-zgzkx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:08.097773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.097753 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-kube-api-access-hrkj8" pod "ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8" (UID: "ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8"). InnerVolumeSpecName "kube-api-access-hrkj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:08.198864 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.198759 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrkj8\" (UniqueName: \"kubernetes.io/projected/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8-kube-api-access-hrkj8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:08.455283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.455199 4969 generic.go:356] "Generic (PLEG): container finished" podID="b115e833-ff1c-4d0a-be5f-13126f600f21" containerID="b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b" exitCode=0 Oct 31 00:41:08.455283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.455251 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" Oct 31 00:41:08.455486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.455275 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" event={"ID":"b115e833-ff1c-4d0a-be5f-13126f600f21","Type":"ContainerDied","Data":"b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b"} Oct 31 00:41:08.455486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.455332 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t" event={"ID":"b115e833-ff1c-4d0a-be5f-13126f600f21","Type":"ContainerDied","Data":"9602df7690331dcb543b88b80b4d0645fbc6955dcec9e9300b7248208f9bbaea"} Oct 31 00:41:08.455486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.455354 4969 scope.go:122] "RemoveContainer" containerID="b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b" Oct 31 00:41:08.457147 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.457124 4969 generic.go:356] "Generic (PLEG): container finished" podID="ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8" containerID="faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21" exitCode=0 Oct 31 00:41:08.457257 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.457180 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" Oct 31 00:41:08.457819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.457237 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" event={"ID":"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8","Type":"ContainerDied","Data":"faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21"} Oct 31 00:41:08.457973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.457860 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l" event={"ID":"ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8","Type":"ContainerDied","Data":"cf9f61aec91137393c640f5cdfb40d85d8ebb4ad0d930d7aab789f7c06cc88d8"} Oct 31 00:41:08.476636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.476611 4969 scope.go:122] "RemoveContainer" containerID="b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b" Oct 31 00:41:08.476934 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:41:08.476912 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b\": container with ID starting with b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b not found: ID does not exist" containerID="b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b" Oct 31 00:41:08.476996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.476942 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b"} err="failed to get container status \"b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b\": rpc error: code = NotFound desc = could not find container \"b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b\": container with ID starting with b91cea24fe760d5a96826e1a3150fbcef38f67855ce28d486fd80a52daa1923b not found: ID does not exist" Oct 31 00:41:08.476996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.476958 4969 scope.go:122] "RemoveContainer" containerID="faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21" Oct 31 00:41:08.486935 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.486918 4969 scope.go:122] "RemoveContainer" containerID="faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21" Oct 31 00:41:08.487213 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:41:08.487188 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21\": container with ID starting with faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21 not found: ID does not exist" containerID="faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21" Oct 31 00:41:08.487308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.487218 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21"} err="failed to get container status \"faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21\": rpc error: code = NotFound desc = could not find container \"faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21\": container with ID starting with faafddea4990ed9121b5d11763e4dfd7bd642b48591b9323b4d8f874031d7d21 not found: ID does not exist" Oct 31 00:41:08.625218 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.625187 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t"] Oct 31 00:41:08.631181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.631157 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-7457c998b8-t4s2t"] Oct 31 00:41:08.741973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.741889 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l"] Oct 31 00:41:08.745791 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:08.745766 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bba5ed-65fc4b9fdc-b4s5l"] Oct 31 00:41:09.247744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:09.247710 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd"] Oct 31 00:41:09.247977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:09.247953 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" containerID="cri-o://b97bde132c97e01aac4f0677e0ccec9a2643cdd5d33851b63dab8722f47e8f89" gracePeriod=30 Oct 31 00:41:09.735730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:09.735697 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8" path="/var/lib/kubelet/pods/ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8/volumes" Oct 31 00:41:09.736321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:09.736299 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b115e833-ff1c-4d0a-be5f-13126f600f21" path="/var/lib/kubelet/pods/b115e833-ff1c-4d0a-be5f-13126f600f21/volumes" Oct 31 00:41:14.507610 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.507563 4969 generic.go:356] "Generic (PLEG): container finished" podID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerID="b97bde132c97e01aac4f0677e0ccec9a2643cdd5d33851b63dab8722f47e8f89" exitCode=0 Oct 31 00:41:14.507949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.507627 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" event={"ID":"a3ce29c6-7ec2-4b1f-a432-114f56ee0830","Type":"ContainerDied","Data":"b97bde132c97e01aac4f0677e0ccec9a2643cdd5d33851b63dab8722f47e8f89"} Oct 31 00:41:14.507949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.507697 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" event={"ID":"a3ce29c6-7ec2-4b1f-a432-114f56ee0830","Type":"ContainerDied","Data":"b2217240253d0b73c0464cb114cd268923b3af45d7bef50733720436fbbda8c0"} Oct 31 00:41:14.507949 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.507725 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2217240253d0b73c0464cb114cd268923b3af45d7bef50733720436fbbda8c0" Oct 31 00:41:14.526299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.526245 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:41:14.584838 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.584806 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-data\") pod \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " Oct 31 00:41:14.584993 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.584949 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-envoy\") pod \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " Oct 31 00:41:14.585058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.584996 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-socket\") pod \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " Oct 31 00:41:14.585058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585026 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-credential-socket\") pod \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " Oct 31 00:41:14.585166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585050 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-data" pod "a3ce29c6-7ec2-4b1f-a432-114f56ee0830" (UID: "a3ce29c6-7ec2-4b1f-a432-114f56ee0830"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:14.585166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585075 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istiod-ca-cert\") pod \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " Oct 31 00:41:14.585166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585109 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-kube-api-access-gvhd9\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-kube-api-access-gvhd9\") pod \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " Oct 31 00:41:14.585166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585161 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-podinfo\") pod \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " Oct 31 00:41:14.585370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585235 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-token\") pod \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " Oct 31 00:41:14.585370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585297 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-certs\") pod \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\" (UID: \"a3ce29c6-7ec2-4b1f-a432-114f56ee0830\") " Oct 31 00:41:14.585576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585295 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-socket" pod "a3ce29c6-7ec2-4b1f-a432-114f56ee0830" (UID: "a3ce29c6-7ec2-4b1f-a432-114f56ee0830"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:14.585576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585311 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-credential-socket" pod "a3ce29c6-7ec2-4b1f-a432-114f56ee0830" (UID: "a3ce29c6-7ec2-4b1f-a432-114f56ee0830"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:14.585576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585522 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-certs" pod "a3ce29c6-7ec2-4b1f-a432-114f56ee0830" (UID: "a3ce29c6-7ec2-4b1f-a432-114f56ee0830"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:14.585806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585576 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istiod-ca-cert" pod "a3ce29c6-7ec2-4b1f-a432-114f56ee0830" (UID: "a3ce29c6-7ec2-4b1f-a432-114f56ee0830"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:14.585847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585814 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:14.585847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585832 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:14.585847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585842 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:14.585969 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585856 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:14.585969 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.585870 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:14.587498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.587471 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-envoy" pod "a3ce29c6-7ec2-4b1f-a432-114f56ee0830" (UID: "a3ce29c6-7ec2-4b1f-a432-114f56ee0830"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:14.587963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.587942 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-kube-api-access-gvhd9" pod "a3ce29c6-7ec2-4b1f-a432-114f56ee0830" (UID: "a3ce29c6-7ec2-4b1f-a432-114f56ee0830"). InnerVolumeSpecName "kube-api-access-gvhd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:14.588171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.588146 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-podinfo" pod "a3ce29c6-7ec2-4b1f-a432-114f56ee0830" (UID: "a3ce29c6-7ec2-4b1f-a432-114f56ee0830"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:41:14.588253 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.588173 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-token" pod "a3ce29c6-7ec2-4b1f-a432-114f56ee0830" (UID: "a3ce29c6-7ec2-4b1f-a432-114f56ee0830"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:14.687389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.687359 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhd9\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-kube-api-access-gvhd9\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:14.687389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.687390 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:14.687645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.687404 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:14.687645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:14.687416 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ce29c6-7ec2-4b1f-a432-114f56ee0830-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:41:15.516540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:15.516511 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd" Oct 31 00:41:15.686472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:15.686439 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd"] Oct 31 00:41:15.736509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:15.736478 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zvlvd"] Oct 31 00:41:17.735964 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:17.735933 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" path="/var/lib/kubelet/pods/a3ce29c6-7ec2-4b1f-a432-114f56ee0830/volumes" Oct 31 00:41:32.728350 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:41:32.728310 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:41:49.253883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:49.253848 4969 scope.go:122] "RemoveContainer" containerID="b97bde132c97e01aac4f0677e0ccec9a2643cdd5d33851b63dab8722f47e8f89" Oct 31 00:41:59.801638 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:59.801602 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76"] Oct 31 00:41:59.802153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:41:59.801866 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" containerID="cri-o://f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6" gracePeriod=30 Oct 31 00:42:02.461100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.461056 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6"] Oct 31 00:42:02.461608 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.461343 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" podUID="74ff306f-d736-452d-831a-b0a3aa39b3ee" containerName="payload-pre-processing" containerID="cri-o://ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550" gracePeriod=30 Oct 31 00:42:02.729983 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:42:02.729945 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:42:02.743492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.743469 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:42:02.883869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.883833 4969 generic.go:356] "Generic (PLEG): container finished" podID="74ff306f-d736-452d-831a-b0a3aa39b3ee" containerID="ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550" exitCode=0 Oct 31 00:42:02.884051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.883880 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" Oct 31 00:42:02.884051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.883919 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" event={"ID":"74ff306f-d736-452d-831a-b0a3aa39b3ee","Type":"ContainerDied","Data":"ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550"} Oct 31 00:42:02.884051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.883977 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6" event={"ID":"74ff306f-d736-452d-831a-b0a3aa39b3ee","Type":"ContainerDied","Data":"75ede7f7feee8e01fefee1311de26906006468c6b010681c56b0a8c648e973c1"} Oct 31 00:42:02.884051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.884006 4969 scope.go:122] "RemoveContainer" containerID="ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550" Oct 31 00:42:02.895440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.895419 4969 scope.go:122] "RemoveContainer" containerID="ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550" Oct 31 00:42:02.895712 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:42:02.895692 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550\": container with ID starting with ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550 not found: ID does not exist" containerID="ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550" Oct 31 00:42:02.895778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.895721 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550"} err="failed to get container status \"ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550\": rpc error: code = NotFound desc = could not find container \"ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550\": container with ID starting with ff1a9f58b8bdbe86d3f31134e5f674f64cd296556cf5dae547af070f8c28a550 not found: ID does not exist" Oct 31 00:42:02.898132 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.898111 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ff306f-d736-452d-831a-b0a3aa39b3ee-kube-api-access-lg6tk\" (UniqueName: \"kubernetes.io/projected/74ff306f-d736-452d-831a-b0a3aa39b3ee-kube-api-access-lg6tk\") pod \"74ff306f-d736-452d-831a-b0a3aa39b3ee\" (UID: \"74ff306f-d736-452d-831a-b0a3aa39b3ee\") " Oct 31 00:42:02.898221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.898207 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ff306f-d736-452d-831a-b0a3aa39b3ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ff306f-d736-452d-831a-b0a3aa39b3ee-plugins-config-volume\") pod \"74ff306f-d736-452d-831a-b0a3aa39b3ee\" (UID: \"74ff306f-d736-452d-831a-b0a3aa39b3ee\") " Oct 31 00:42:02.898583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.898560 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ff306f-d736-452d-831a-b0a3aa39b3ee-plugins-config-volume" pod "74ff306f-d736-452d-831a-b0a3aa39b3ee" (UID: "74ff306f-d736-452d-831a-b0a3aa39b3ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:02.898911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.898892 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ff306f-d736-452d-831a-b0a3aa39b3ee-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:02.901016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.900987 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ff306f-d736-452d-831a-b0a3aa39b3ee-kube-api-access-lg6tk" pod "74ff306f-d736-452d-831a-b0a3aa39b3ee" (UID: "74ff306f-d736-452d-831a-b0a3aa39b3ee"). InnerVolumeSpecName "kube-api-access-lg6tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:03.000104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:02.999992 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg6tk\" (UniqueName: \"kubernetes.io/projected/74ff306f-d736-452d-831a-b0a3aa39b3ee-kube-api-access-lg6tk\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:03.347027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:03.346945 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6"] Oct 31 00:42:03.365633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:03.365579 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-6bcb6c79cf-gscs6"] Oct 31 00:42:03.735241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:03.735204 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ff306f-d736-452d-831a-b0a3aa39b3ee" path="/var/lib/kubelet/pods/74ff306f-d736-452d-831a-b0a3aa39b3ee/volumes" Oct 31 00:42:05.064550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.064495 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:42:05.125912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.125879 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-credential-socket\") pod \"2a8ad439-95c6-4274-8577-4719de10fb3f\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " Oct 31 00:42:05.126108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.125931 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8ad439-95c6-4274-8577-4719de10fb3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8ad439-95c6-4274-8577-4719de10fb3f-istiod-ca-cert\") pod \"2a8ad439-95c6-4274-8577-4719de10fb3f\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " Oct 31 00:42:05.126108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.125984 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-certs\") pod \"2a8ad439-95c6-4274-8577-4719de10fb3f\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " Oct 31 00:42:05.126108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126012 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-token\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-token\") pod \"2a8ad439-95c6-4274-8577-4719de10fb3f\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " Oct 31 00:42:05.126108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126051 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-envoy\") pod \"2a8ad439-95c6-4274-8577-4719de10fb3f\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " Oct 31 00:42:05.126343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126137 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-data\") pod \"2a8ad439-95c6-4274-8577-4719de10fb3f\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " Oct 31 00:42:05.126343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126173 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-socket\") pod \"2a8ad439-95c6-4274-8577-4719de10fb3f\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " Oct 31 00:42:05.126343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126270 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-podinfo\") pod \"2a8ad439-95c6-4274-8577-4719de10fb3f\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " Oct 31 00:42:05.126343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126271 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-credential-socket" pod "2a8ad439-95c6-4274-8577-4719de10fb3f" (UID: "2a8ad439-95c6-4274-8577-4719de10fb3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.126343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126332 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-kube-api-access-m7nf5\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-kube-api-access-m7nf5\") pod \"2a8ad439-95c6-4274-8577-4719de10fb3f\" (UID: \"2a8ad439-95c6-4274-8577-4719de10fb3f\") " Oct 31 00:42:05.126632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126358 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-certs" pod "2a8ad439-95c6-4274-8577-4719de10fb3f" (UID: "2a8ad439-95c6-4274-8577-4719de10fb3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.126632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126485 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8ad439-95c6-4274-8577-4719de10fb3f-istiod-ca-cert" pod "2a8ad439-95c6-4274-8577-4719de10fb3f" (UID: "2a8ad439-95c6-4274-8577-4719de10fb3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:05.126632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126578 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-socket" pod "2a8ad439-95c6-4274-8577-4719de10fb3f" (UID: "2a8ad439-95c6-4274-8577-4719de10fb3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.126765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.126739 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-data" pod "2a8ad439-95c6-4274-8577-4719de10fb3f" (UID: "2a8ad439-95c6-4274-8577-4719de10fb3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.127225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.127204 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.127280 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.127232 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8ad439-95c6-4274-8577-4719de10fb3f-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.127280 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.127248 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.127280 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.127263 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.127380 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.127277 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.128516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.128493 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-envoy" pod "2a8ad439-95c6-4274-8577-4719de10fb3f" (UID: "2a8ad439-95c6-4274-8577-4719de10fb3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:05.128705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.128688 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-token" pod "2a8ad439-95c6-4274-8577-4719de10fb3f" (UID: "2a8ad439-95c6-4274-8577-4719de10fb3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:05.128787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.128771 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-kube-api-access-m7nf5" pod "2a8ad439-95c6-4274-8577-4719de10fb3f" (UID: "2a8ad439-95c6-4274-8577-4719de10fb3f"). InnerVolumeSpecName "kube-api-access-m7nf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:05.129315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.129294 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-podinfo" pod "2a8ad439-95c6-4274-8577-4719de10fb3f" (UID: "2a8ad439-95c6-4274-8577-4719de10fb3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:05.228472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.228440 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.228472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.228472 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.228712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.228490 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8ad439-95c6-4274-8577-4719de10fb3f-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.228712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.228505 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7nf5\" (UniqueName: \"kubernetes.io/projected/2a8ad439-95c6-4274-8577-4719de10fb3f-kube-api-access-m7nf5\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:05.909479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.909439 4969 generic.go:356] "Generic (PLEG): container finished" podID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerID="f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6" exitCode=0 Oct 31 00:42:05.909705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.909511 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" Oct 31 00:42:05.909705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.909523 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" event={"ID":"2a8ad439-95c6-4274-8577-4719de10fb3f","Type":"ContainerDied","Data":"f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6"} Oct 31 00:42:05.909705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.909583 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76" event={"ID":"2a8ad439-95c6-4274-8577-4719de10fb3f","Type":"ContainerDied","Data":"5d2a6fac470a9ce3baabcc0fc064e478dc2f182c724c08a5a1cea7298c9f39e6"} Oct 31 00:42:05.909705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.909620 4969 scope.go:122] "RemoveContainer" containerID="f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6" Oct 31 00:42:05.921134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.920959 4969 scope.go:122] "RemoveContainer" containerID="66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8" Oct 31 00:42:05.932081 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.932061 4969 scope.go:122] "RemoveContainer" containerID="f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6" Oct 31 00:42:05.932394 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:42:05.932368 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6\": container with ID starting with f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6 not found: ID does not exist" containerID="f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6" Oct 31 00:42:05.932469 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.932404 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6"} err="failed to get container status \"f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6\": rpc error: code = NotFound desc = could not find container \"f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6\": container with ID starting with f33c7d4524a6ac1cbe4882647e0b20555e454267b4ce0c231395cae6c73290c6 not found: ID does not exist" Oct 31 00:42:05.932469 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.932424 4969 scope.go:122] "RemoveContainer" containerID="66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8" Oct 31 00:42:05.932706 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:42:05.932686 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8\": container with ID starting with 66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8 not found: ID does not exist" containerID="66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8" Oct 31 00:42:05.932759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:05.932712 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8"} err="failed to get container status \"66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8\": rpc error: code = NotFound desc = could not find container \"66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8\": container with ID starting with 66f9c3812c1967274f65a3b57639b1df28218a0c7783dde791f2cd37f5fd4ab8 not found: ID does not exist" Oct 31 00:42:06.126018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:06.125983 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76"] Oct 31 00:42:06.143667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:06.143636 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bba5ed-openshift-default-587c8c7bf-qwq76"] Oct 31 00:42:07.735381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:07.735335 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" path="/var/lib/kubelet/pods/2a8ad439-95c6-4274-8577-4719de10fb3f/volumes" Oct 31 00:42:32.728602 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:42:32.728551 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:42:37.121524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.121490 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd"] Oct 31 00:42:37.122093 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.121743 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" podUID="29f60d2c-d53b-4df2-a835-99f00659294a" containerName="maas-api" containerID="cri-o://44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026" gracePeriod=30 Oct 31 00:42:37.425179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.425153 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:42:37.527504 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.527431 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f60d2c-d53b-4df2-a835-99f00659294a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f60d2c-d53b-4df2-a835-99f00659294a-openshift-service-ca\") pod \"29f60d2c-d53b-4df2-a835-99f00659294a\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " Oct 31 00:42:37.527729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.527566 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls\") pod \"29f60d2c-d53b-4df2-a835-99f00659294a\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " Oct 31 00:42:37.527729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.527644 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-metrics-certs\") pod \"29f60d2c-d53b-4df2-a835-99f00659294a\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " Oct 31 00:42:37.527729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.527718 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f60d2c-d53b-4df2-a835-99f00659294a-kube-api-access-k7jwh\" (UniqueName: \"kubernetes.io/projected/29f60d2c-d53b-4df2-a835-99f00659294a-kube-api-access-k7jwh\") pod \"29f60d2c-d53b-4df2-a835-99f00659294a\" (UID: \"29f60d2c-d53b-4df2-a835-99f00659294a\") " Oct 31 00:42:37.528006 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.527939 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f60d2c-d53b-4df2-a835-99f00659294a-openshift-service-ca" pod "29f60d2c-d53b-4df2-a835-99f00659294a" (UID: "29f60d2c-d53b-4df2-a835-99f00659294a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:37.528703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.528641 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f60d2c-d53b-4df2-a835-99f00659294a-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:37.531314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.531284 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-metrics-certs" pod "29f60d2c-d53b-4df2-a835-99f00659294a" (UID: "29f60d2c-d53b-4df2-a835-99f00659294a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:37.532185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.532159 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f60d2c-d53b-4df2-a835-99f00659294a-kube-api-access-k7jwh" pod "29f60d2c-d53b-4df2-a835-99f00659294a" (UID: "29f60d2c-d53b-4df2-a835-99f00659294a"). InnerVolumeSpecName "kube-api-access-k7jwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:37.532532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.532485 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls" pod "29f60d2c-d53b-4df2-a835-99f00659294a" (UID: "29f60d2c-d53b-4df2-a835-99f00659294a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:37.608040 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.608009 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl"] Oct 31 00:42:37.608302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.608259 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" podUID="70ecefa7-43bc-4a70-950e-2e28d2dc6e43" containerName="payload-processing" containerID="cri-o://16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61" gracePeriod=30 Oct 31 00:42:37.629480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.629458 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:37.629573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.629483 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f60d2c-d53b-4df2-a835-99f00659294a-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:37.629573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.629496 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7jwh\" (UniqueName: \"kubernetes.io/projected/29f60d2c-d53b-4df2-a835-99f00659294a-kube-api-access-k7jwh\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:37.649558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.649505 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4"] Oct 31 00:42:37.649728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.649708 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" podUID="3a2c7172-53f4-42f9-8966-6aa877920ae2" containerName="payload-pre-processing" containerID="cri-o://371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e" gracePeriod=30 Oct 31 00:42:37.705077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.705043 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" podUID="3a2c7172-53f4-42f9-8966-6aa877920ae2" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.170:9004: connect: connection refused" Oct 31 00:42:37.904418 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.904356 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:42:37.941541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:37.941517 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:42:38.034625 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.034578 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-plugins-config-volume\") pod \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\" (UID: \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\") " Oct 31 00:42:38.034798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.034716 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-kube-api-access-gdxnm\" (UniqueName: \"kubernetes.io/projected/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-kube-api-access-gdxnm\") pod \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\" (UID: \"70ecefa7-43bc-4a70-950e-2e28d2dc6e43\") " Oct 31 00:42:38.034798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.034751 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a2c7172-53f4-42f9-8966-6aa877920ae2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2c7172-53f4-42f9-8966-6aa877920ae2-plugins-config-volume\") pod \"3a2c7172-53f4-42f9-8966-6aa877920ae2\" (UID: \"3a2c7172-53f4-42f9-8966-6aa877920ae2\") " Oct 31 00:42:38.034897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.034882 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a2c7172-53f4-42f9-8966-6aa877920ae2-kube-api-access-rl7bq\" (UniqueName: \"kubernetes.io/projected/3a2c7172-53f4-42f9-8966-6aa877920ae2-kube-api-access-rl7bq\") pod \"3a2c7172-53f4-42f9-8966-6aa877920ae2\" (UID: \"3a2c7172-53f4-42f9-8966-6aa877920ae2\") " Oct 31 00:42:38.035062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.035035 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-plugins-config-volume" pod "70ecefa7-43bc-4a70-950e-2e28d2dc6e43" (UID: "70ecefa7-43bc-4a70-950e-2e28d2dc6e43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:38.035265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.035236 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a2c7172-53f4-42f9-8966-6aa877920ae2-plugins-config-volume" pod "3a2c7172-53f4-42f9-8966-6aa877920ae2" (UID: "3a2c7172-53f4-42f9-8966-6aa877920ae2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:38.035435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.035421 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2c7172-53f4-42f9-8966-6aa877920ae2-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:38.035485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.035438 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:38.037113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.037093 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-kube-api-access-gdxnm" pod "70ecefa7-43bc-4a70-950e-2e28d2dc6e43" (UID: "70ecefa7-43bc-4a70-950e-2e28d2dc6e43"). InnerVolumeSpecName "kube-api-access-gdxnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:38.037113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.037103 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2c7172-53f4-42f9-8966-6aa877920ae2-kube-api-access-rl7bq" pod "3a2c7172-53f4-42f9-8966-6aa877920ae2" (UID: "3a2c7172-53f4-42f9-8966-6aa877920ae2"). InnerVolumeSpecName "kube-api-access-rl7bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:38.151957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.151920 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl7bq\" (UniqueName: \"kubernetes.io/projected/3a2c7172-53f4-42f9-8966-6aa877920ae2-kube-api-access-rl7bq\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:38.151957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.151953 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdxnm\" (UniqueName: \"kubernetes.io/projected/70ecefa7-43bc-4a70-950e-2e28d2dc6e43-kube-api-access-gdxnm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:38.180713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.180100 4969 generic.go:356] "Generic (PLEG): container finished" podID="70ecefa7-43bc-4a70-950e-2e28d2dc6e43" containerID="16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61" exitCode=0 Oct 31 00:42:38.180713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.180157 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" Oct 31 00:42:38.180713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.180195 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" event={"ID":"70ecefa7-43bc-4a70-950e-2e28d2dc6e43","Type":"ContainerDied","Data":"16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61"} Oct 31 00:42:38.180713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.180268 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl" event={"ID":"70ecefa7-43bc-4a70-950e-2e28d2dc6e43","Type":"ContainerDied","Data":"7fee01e7580c166f2fdae057d2b1fcf498b96736ece83ba37d2b7d636ac3c590"} Oct 31 00:42:38.180713 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.180292 4969 scope.go:122] "RemoveContainer" containerID="16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61" Oct 31 00:42:38.184880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.184811 4969 generic.go:356] "Generic (PLEG): container finished" podID="29f60d2c-d53b-4df2-a835-99f00659294a" containerID="44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026" exitCode=0 Oct 31 00:42:38.185168 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.185146 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" event={"ID":"29f60d2c-d53b-4df2-a835-99f00659294a","Type":"ContainerDied","Data":"44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026"} Oct 31 00:42:38.185323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.185303 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" event={"ID":"29f60d2c-d53b-4df2-a835-99f00659294a","Type":"ContainerDied","Data":"b73cf3df65fcf720467bcf598a9310f0b4a8d963e02de13b38dd132f5beecc7c"} Oct 31 00:42:38.185530 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.185504 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd" Oct 31 00:42:38.188191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.188170 4969 generic.go:356] "Generic (PLEG): container finished" podID="3a2c7172-53f4-42f9-8966-6aa877920ae2" containerID="371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e" exitCode=0 Oct 31 00:42:38.188291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.188206 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" event={"ID":"3a2c7172-53f4-42f9-8966-6aa877920ae2","Type":"ContainerDied","Data":"371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e"} Oct 31 00:42:38.188291 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.188241 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" event={"ID":"3a2c7172-53f4-42f9-8966-6aa877920ae2","Type":"ContainerDied","Data":"d65ee17e7465011e7822fcd3bc9bdcf075d2da4131d43eee0699b65570e3b180"} Oct 31 00:42:38.188423 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.188297 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4" Oct 31 00:42:38.200660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.200428 4969 scope.go:122] "RemoveContainer" containerID="16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61" Oct 31 00:42:38.202637 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:42:38.201106 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61\": container with ID starting with 16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61 not found: ID does not exist" containerID="16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61" Oct 31 00:42:38.202637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.201146 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61"} err="failed to get container status \"16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61\": rpc error: code = NotFound desc = could not find container \"16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61\": container with ID starting with 16afa7b5195f39e9e9296ff8da6a87765344ed5629ac88c4679fc429d1b28b61 not found: ID does not exist" Oct 31 00:42:38.202637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.201170 4969 scope.go:122] "RemoveContainer" containerID="44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026" Oct 31 00:42:38.212509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.212493 4969 scope.go:122] "RemoveContainer" containerID="44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026" Oct 31 00:42:38.212813 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:42:38.212795 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026\": container with ID starting with 44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026 not found: ID does not exist" containerID="44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026" Oct 31 00:42:38.212918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.212824 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026"} err="failed to get container status \"44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026\": rpc error: code = NotFound desc = could not find container \"44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026\": container with ID starting with 44772eec00b5ed9fb6577ea675411d019f57b9dd76f5e0aedba0de9cebd4e026 not found: ID does not exist" Oct 31 00:42:38.212918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.212856 4969 scope.go:122] "RemoveContainer" containerID="371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e" Oct 31 00:42:38.222691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.222670 4969 scope.go:122] "RemoveContainer" containerID="371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e" Oct 31 00:42:38.222963 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:42:38.222945 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e\": container with ID starting with 371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e not found: ID does not exist" containerID="371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e" Oct 31 00:42:38.223030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.222972 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e"} err="failed to get container status \"371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e\": rpc error: code = NotFound desc = could not find container \"371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e\": container with ID starting with 371f6116f157bbc8c5696a60334bddf593dba35e122b86bfe638b5687c08c00e not found: ID does not exist" Oct 31 00:42:38.348192 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.348158 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4"] Oct 31 00:42:38.358515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.358484 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-70babe-78d784d879-c7mn4"] Oct 31 00:42:38.468300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.468218 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd"] Oct 31 00:42:38.473310 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.473286 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-70babe-64cd7c557f-4jmhd"] Oct 31 00:42:38.584100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.584060 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl"] Oct 31 00:42:38.593485 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:38.593453 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-70babe-cb57d9d6-ppxxl"] Oct 31 00:42:39.735901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:39.735872 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29f60d2c-d53b-4df2-a835-99f00659294a" path="/var/lib/kubelet/pods/29f60d2c-d53b-4df2-a835-99f00659294a/volumes" Oct 31 00:42:39.736285 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:39.736244 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a2c7172-53f4-42f9-8966-6aa877920ae2" path="/var/lib/kubelet/pods/3a2c7172-53f4-42f9-8966-6aa877920ae2/volumes" Oct 31 00:42:39.736620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:39.736605 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ecefa7-43bc-4a70-950e-2e28d2dc6e43" path="/var/lib/kubelet/pods/70ecefa7-43bc-4a70-950e-2e28d2dc6e43/volumes" Oct 31 00:42:44.938886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.938851 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss"] Oct 31 00:42:44.939444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939429 4969 cpu_manager.go:414] "containerMap: removing container" podUID="66768b27-3dd7-4b05-a1e7-e936dccb7e7c" containerName="payload-pre-processing" Oct 31 00:42:44.939489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939445 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66768b27-3dd7-4b05-a1e7-e936dccb7e7c" containerName="payload-pre-processing" Oct 31 00:42:44.939489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939468 4969 cpu_manager.go:414] "containerMap: removing container" podUID="29f60d2c-d53b-4df2-a835-99f00659294a" containerName="maas-api" Oct 31 00:42:44.939489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939476 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f60d2c-d53b-4df2-a835-99f00659294a" containerName="maas-api" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939494 4969 cpu_manager.go:414] "containerMap: removing container" podUID="70ecefa7-43bc-4a70-950e-2e28d2dc6e43" containerName="payload-processing" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939499 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ecefa7-43bc-4a70-950e-2e28d2dc6e43" containerName="payload-processing" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939511 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b115e833-ff1c-4d0a-be5f-13126f600f21" containerName="payload-processing" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939516 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b115e833-ff1c-4d0a-be5f-13126f600f21" containerName="payload-processing" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939524 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939529 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939536 4969 cpu_manager.go:414] "containerMap: removing container" podUID="3a2c7172-53f4-42f9-8966-6aa877920ae2" containerName="payload-pre-processing" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939541 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a2c7172-53f4-42f9-8966-6aa877920ae2" containerName="payload-pre-processing" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939555 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939559 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939569 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939573 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939580 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8" containerName="payload-pre-processing" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939584 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8" containerName="payload-pre-processing" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939603 4969 cpu_manager.go:414] "containerMap: removing container" podUID="74ff306f-d736-452d-831a-b0a3aa39b3ee" containerName="payload-pre-processing" Oct 31 00:42:44.939584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939608 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ff306f-d736-452d-831a-b0a3aa39b3ee" containerName="payload-pre-processing" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939706 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ff306f-d736-452d-831a-b0a3aa39b3ee" containerName="payload-pre-processing" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939716 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="66768b27-3dd7-4b05-a1e7-e936dccb7e7c" containerName="payload-pre-processing" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939724 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f60d2c-d53b-4df2-a835-99f00659294a" containerName="maas-api" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939731 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ecefa7-43bc-4a70-950e-2e28d2dc6e43" containerName="payload-processing" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939737 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b115e833-ff1c-4d0a-be5f-13126f600f21" containerName="payload-processing" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939743 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ce29c6-7ec2-4b1f-a432-114f56ee0830" containerName="istio-proxy" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939748 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a2c7172-53f4-42f9-8966-6aa877920ae2" containerName="payload-pre-processing" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939756 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939763 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8ad439-95c6-4274-8577-4719de10fb3f" containerName="istio-proxy" Oct 31 00:42:44.940129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.939768 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab44c7a3-7f05-4e38-838f-0e8a9ee4c5c8" containerName="payload-pre-processing" Oct 31 00:42:44.943189 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:44.943173 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:45.038496 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.038459 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2318af60-6336-43f6-a954-33f0cc497337-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss\" (UID: \"2318af60-6336-43f6-a954-33f0cc497337\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:45.038703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.038678 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrqd\" (UniqueName: \"kubernetes.io/projected/2318af60-6336-43f6-a954-33f0cc497337-kube-api-access-pvrqd\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss\" (UID: \"2318af60-6336-43f6-a954-33f0cc497337\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:45.046135 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.046106 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a4f30\"" Oct 31 00:42:45.046261 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.046245 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a4f30-dockercfg-6hwbh\"" Oct 31 00:42:45.049678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.049658 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss"] Oct 31 00:42:45.140372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.140313 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrqd\" (UniqueName: \"kubernetes.io/projected/2318af60-6336-43f6-a954-33f0cc497337-kube-api-access-pvrqd\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss\" (UID: \"2318af60-6336-43f6-a954-33f0cc497337\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:45.140542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.140525 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2318af60-6336-43f6-a954-33f0cc497337-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss\" (UID: \"2318af60-6336-43f6-a954-33f0cc497337\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:45.141250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.141229 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2318af60-6336-43f6-a954-33f0cc497337-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss\" (UID: \"2318af60-6336-43f6-a954-33f0cc497337\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:45.204703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.204622 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrqd\" (UniqueName: \"kubernetes.io/projected/2318af60-6336-43f6-a954-33f0cc497337-kube-api-access-pvrqd\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss\" (UID: \"2318af60-6336-43f6-a954-33f0cc497337\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:45.257067 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.257035 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:45.656181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:45.656154 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss"] Oct 31 00:42:45.657718 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:42:45.657691 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2318af60_6336_43f6_a954_33f0cc497337.slice/crio-390ce14463d63d941a742eadf33b0be2ba6934968a0b54df2f87544a5e220913 WatchSource:0}: Error finding container 390ce14463d63d941a742eadf33b0be2ba6934968a0b54df2f87544a5e220913: Status 404 returned error can't find the container with id 390ce14463d63d941a742eadf33b0be2ba6934968a0b54df2f87544a5e220913 Oct 31 00:42:46.259466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:46.259430 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" event={"ID":"2318af60-6336-43f6-a954-33f0cc497337","Type":"ContainerStarted","Data":"db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba"} Oct 31 00:42:46.259900 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:46.259499 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:46.259900 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:46.259521 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" event={"ID":"2318af60-6336-43f6-a954-33f0cc497337","Type":"ContainerStarted","Data":"390ce14463d63d941a742eadf33b0be2ba6934968a0b54df2f87544a5e220913"} Oct 31 00:42:46.384547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:46.384475 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" podStartSLOduration=2.384457005 podStartE2EDuration="2.384457005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:46.383907978 +0000 UTC m=+2398.859266591" watchObservedRunningTime="2026-10-31 00:42:46.384457005 +0000 UTC m=+2398.859815429" Oct 31 00:42:47.201900 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:47.201857 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj"] Oct 31 00:42:47.273970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:47.273770 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" containerID="cri-o://fe3022a942ef3bbe1786e33a5736bf3a69431635547197ca86e12a6277ecbad7" gracePeriod=30 Oct 31 00:42:48.138712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:48.138685 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:42:48.146906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:48.146880 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:42:48.154353 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:48.154335 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:42:48.178144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:48.178113 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:42:49.692100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.692066 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k"] Oct 31 00:42:49.697830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.697809 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.752808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.752776 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-38dd3e-openshift-default-dockercfg-gt9m2\"" Oct 31 00:42:49.795909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.795855 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-credential-socket\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.795909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.795912 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713e3d53-958f-4f7a-bdbb-645c326011e4-istiod-ca-cert\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.796120 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.796011 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-podinfo\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.796120 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.796052 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-certs\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.796221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.796149 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5l6\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-kube-api-access-6f5l6\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.796221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.796177 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-token\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.796361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.796227 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-envoy\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.796361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.796298 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-data\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.796438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.796400 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-socket\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.807627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.807576 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k"] Oct 31 00:42:49.897184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897145 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-credential-socket\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.897184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897188 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713e3d53-958f-4f7a-bdbb-645c326011e4-istiod-ca-cert\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.897438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897241 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-podinfo\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.897438 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897271 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-certs\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.897553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897504 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5l6\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-kube-api-access-6f5l6\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.897553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897542 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-token\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.897702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897636 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-envoy\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.897702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897663 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-data\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.897702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897667 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-credential-socket\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.897839 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.897758 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-socket\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.898127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.898102 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-certs\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.898358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.898336 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-socket\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.898448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.898379 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713e3d53-958f-4f7a-bdbb-645c326011e4-istiod-ca-cert\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.898448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.898401 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-data\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.900183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.900152 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-podinfo\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.900274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.900206 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-envoy\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.958738 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.958675 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-token\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:49.963461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:49.963434 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5l6\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-kube-api-access-6f5l6\") pod \"e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:50.011832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:50.011794 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:50.453148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:50.453120 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k"] Oct 31 00:42:50.485642 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:42:50.485613 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713e3d53_958f_4f7a_bdbb_645c326011e4.slice/crio-574bbe8eae777e059b22159f6a34ffc2ce667e9ab3ab096b28e761c2f4e15c61 WatchSource:0}: Error finding container 574bbe8eae777e059b22159f6a34ffc2ce667e9ab3ab096b28e761c2f4e15c61: Status 404 returned error can't find the container with id 574bbe8eae777e059b22159f6a34ffc2ce667e9ab3ab096b28e761c2f4e15c61 Oct 31 00:42:50.488326 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:50.488296 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:42:50.488397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:50.488363 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:42:50.488397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:50.488392 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:42:51.302850 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:51.302813 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" event={"ID":"713e3d53-958f-4f7a-bdbb-645c326011e4","Type":"ContainerStarted","Data":"407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166"} Oct 31 00:42:51.303252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:51.302907 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" event={"ID":"713e3d53-958f-4f7a-bdbb-645c326011e4","Type":"ContainerStarted","Data":"574bbe8eae777e059b22159f6a34ffc2ce667e9ab3ab096b28e761c2f4e15c61"} Oct 31 00:42:51.480723 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:51.480674 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" podStartSLOduration=2.4806593169999998 podStartE2EDuration="2.480659317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:51.47619208 +0000 UTC m=+2403.951550504" watchObservedRunningTime="2026-10-31 00:42:51.480659317 +0000 UTC m=+2403.956017747" Oct 31 00:42:52.017650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:52.012953 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:52.023846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:52.023816 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:52.311607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:52.311561 4969 generic.go:356] "Generic (PLEG): container finished" podID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerID="fe3022a942ef3bbe1786e33a5736bf3a69431635547197ca86e12a6277ecbad7" exitCode=0 Oct 31 00:42:52.312047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:52.311640 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" event={"ID":"13f2becb-a214-43c1-8d2c-519a00bfd076","Type":"ContainerDied","Data":"fe3022a942ef3bbe1786e33a5736bf3a69431635547197ca86e12a6277ecbad7"} Oct 31 00:42:52.312047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:52.311711 4969 scope.go:122] "RemoveContainer" containerID="81464f271916c299164d58c66e6365530f0aeae425794f54e7375d823e51096c" Oct 31 00:42:52.312047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:52.311935 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:52.313166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:52.313148 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:42:52.965025 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:52.964998 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:42:53.033743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.033707 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-socket\") pod \"13f2becb-a214-43c1-8d2c-519a00bfd076\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " Oct 31 00:42:53.033934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.033773 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-kube-api-access-7c8l8\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-kube-api-access-7c8l8\") pod \"13f2becb-a214-43c1-8d2c-519a00bfd076\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " Oct 31 00:42:53.033934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.033812 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-data\") pod \"13f2becb-a214-43c1-8d2c-519a00bfd076\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " Oct 31 00:42:53.033934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.033850 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f2becb-a214-43c1-8d2c-519a00bfd076-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f2becb-a214-43c1-8d2c-519a00bfd076-istiod-ca-cert\") pod \"13f2becb-a214-43c1-8d2c-519a00bfd076\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " Oct 31 00:42:53.033934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.033919 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-certs\") pod \"13f2becb-a214-43c1-8d2c-519a00bfd076\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " Oct 31 00:42:53.034171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.033949 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-envoy\") pod \"13f2becb-a214-43c1-8d2c-519a00bfd076\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " Oct 31 00:42:53.034171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034012 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-token\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-token\") pod \"13f2becb-a214-43c1-8d2c-519a00bfd076\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " Oct 31 00:42:53.034171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034065 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-podinfo\") pod \"13f2becb-a214-43c1-8d2c-519a00bfd076\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " Oct 31 00:42:53.034171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034125 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-data" pod "13f2becb-a214-43c1-8d2c-519a00bfd076" (UID: "13f2becb-a214-43c1-8d2c-519a00bfd076"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:53.034171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034130 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-socket" pod "13f2becb-a214-43c1-8d2c-519a00bfd076" (UID: "13f2becb-a214-43c1-8d2c-519a00bfd076"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:53.034171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034132 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-credential-socket\") pod \"13f2becb-a214-43c1-8d2c-519a00bfd076\" (UID: \"13f2becb-a214-43c1-8d2c-519a00bfd076\") " Oct 31 00:42:53.034495 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034330 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-certs" pod "13f2becb-a214-43c1-8d2c-519a00bfd076" (UID: "13f2becb-a214-43c1-8d2c-519a00bfd076"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:53.034495 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034389 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-credential-socket" pod "13f2becb-a214-43c1-8d2c-519a00bfd076" (UID: "13f2becb-a214-43c1-8d2c-519a00bfd076"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:53.034495 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034411 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f2becb-a214-43c1-8d2c-519a00bfd076-istiod-ca-cert" pod "13f2becb-a214-43c1-8d2c-519a00bfd076" (UID: "13f2becb-a214-43c1-8d2c-519a00bfd076"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:53.034841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034767 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.034841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034791 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.034841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034802 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f2becb-a214-43c1-8d2c-519a00bfd076-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.034841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034811 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.034841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.034820 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.036424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.036398 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-kube-api-access-7c8l8" pod "13f2becb-a214-43c1-8d2c-519a00bfd076" (UID: "13f2becb-a214-43c1-8d2c-519a00bfd076"). InnerVolumeSpecName "kube-api-access-7c8l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:53.036529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.036426 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-token" pod "13f2becb-a214-43c1-8d2c-519a00bfd076" (UID: "13f2becb-a214-43c1-8d2c-519a00bfd076"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:53.036529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.036443 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-podinfo" pod "13f2becb-a214-43c1-8d2c-519a00bfd076" (UID: "13f2becb-a214-43c1-8d2c-519a00bfd076"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:53.036979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.036955 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-envoy" pod "13f2becb-a214-43c1-8d2c-519a00bfd076" (UID: "13f2becb-a214-43c1-8d2c-519a00bfd076"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:53.135908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.135823 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.135908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.135855 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.135908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.135869 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f2becb-a214-43c1-8d2c-519a00bfd076-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.135908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.135882 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8l8\" (UniqueName: \"kubernetes.io/projected/13f2becb-a214-43c1-8d2c-519a00bfd076-kube-api-access-7c8l8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.320569 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.320485 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" Oct 31 00:42:53.320569 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.320524 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj" event={"ID":"13f2becb-a214-43c1-8d2c-519a00bfd076","Type":"ContainerDied","Data":"eff7ef4a12874002c3f92f93d5e2a37d4e0ce2cb4bdd53a43082a5f7144627c9"} Oct 31 00:42:53.321148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.320582 4969 scope.go:122] "RemoveContainer" containerID="fe3022a942ef3bbe1786e33a5736bf3a69431635547197ca86e12a6277ecbad7" Oct 31 00:42:53.487173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.487146 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj"] Oct 31 00:42:53.499769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.499744 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-70babe-openshift-default-664c65585f-2zskj"] Oct 31 00:42:53.735488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:53.735454 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" path="/var/lib/kubelet/pods/13f2becb-a214-43c1-8d2c-519a00bfd076/volumes" Oct 31 00:42:56.198326 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.198281 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq"] Oct 31 00:42:56.198764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.198507 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" podUID="1235cae5-9e24-46ef-879a-4c88ff2220cb" containerName="maas-api" containerID="cri-o://46e60a74c197b19816e179c1eaa101bed33bcdc164f1c7308dc946093ab30c7f" gracePeriod=30 Oct 31 00:42:56.357058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.357028 4969 generic.go:356] "Generic (PLEG): container finished" podID="1235cae5-9e24-46ef-879a-4c88ff2220cb" containerID="46e60a74c197b19816e179c1eaa101bed33bcdc164f1c7308dc946093ab30c7f" exitCode=0 Oct 31 00:42:56.357212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.357106 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" event={"ID":"1235cae5-9e24-46ef-879a-4c88ff2220cb","Type":"ContainerDied","Data":"46e60a74c197b19816e179c1eaa101bed33bcdc164f1c7308dc946093ab30c7f"} Oct 31 00:42:56.483024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.482999 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:42:56.575039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.575009 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-metrics-certs\") pod \"1235cae5-9e24-46ef-879a-4c88ff2220cb\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " Oct 31 00:42:56.575183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.575045 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1235cae5-9e24-46ef-879a-4c88ff2220cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235cae5-9e24-46ef-879a-4c88ff2220cb-openshift-service-ca\") pod \"1235cae5-9e24-46ef-879a-4c88ff2220cb\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " Oct 31 00:42:56.575183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.575147 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1235cae5-9e24-46ef-879a-4c88ff2220cb-kube-api-access-cfnhg\" (UniqueName: \"kubernetes.io/projected/1235cae5-9e24-46ef-879a-4c88ff2220cb-kube-api-access-cfnhg\") pod \"1235cae5-9e24-46ef-879a-4c88ff2220cb\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " Oct 31 00:42:56.575304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.575244 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls\") pod \"1235cae5-9e24-46ef-879a-4c88ff2220cb\" (UID: \"1235cae5-9e24-46ef-879a-4c88ff2220cb\") " Oct 31 00:42:56.575451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.575423 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1235cae5-9e24-46ef-879a-4c88ff2220cb-openshift-service-ca" pod "1235cae5-9e24-46ef-879a-4c88ff2220cb" (UID: "1235cae5-9e24-46ef-879a-4c88ff2220cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:56.575816 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.575793 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1235cae5-9e24-46ef-879a-4c88ff2220cb-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:56.577408 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.577384 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-metrics-certs" pod "1235cae5-9e24-46ef-879a-4c88ff2220cb" (UID: "1235cae5-9e24-46ef-879a-4c88ff2220cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:56.577562 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.577540 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls" pod "1235cae5-9e24-46ef-879a-4c88ff2220cb" (UID: "1235cae5-9e24-46ef-879a-4c88ff2220cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:56.577660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.577554 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1235cae5-9e24-46ef-879a-4c88ff2220cb-kube-api-access-cfnhg" pod "1235cae5-9e24-46ef-879a-4c88ff2220cb" (UID: "1235cae5-9e24-46ef-879a-4c88ff2220cb"). InnerVolumeSpecName "kube-api-access-cfnhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:56.677114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.677081 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfnhg\" (UniqueName: \"kubernetes.io/projected/1235cae5-9e24-46ef-879a-4c88ff2220cb-kube-api-access-cfnhg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:56.677114 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.677113 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:56.677313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:56.677122 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1235cae5-9e24-46ef-879a-4c88ff2220cb-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.275337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:57.275310 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:42:57.365920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:57.365891 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" Oct 31 00:42:57.366086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:57.365919 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq" event={"ID":"1235cae5-9e24-46ef-879a-4c88ff2220cb","Type":"ContainerDied","Data":"f95067d49b6bea3e2a7ac301651996213bd2c1806cb39e5f85076252c3d253d5"} Oct 31 00:42:57.366086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:57.365984 4969 scope.go:122] "RemoveContainer" containerID="46e60a74c197b19816e179c1eaa101bed33bcdc164f1c7308dc946093ab30c7f" Oct 31 00:42:57.511785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:57.511752 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq"] Oct 31 00:42:57.529542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:57.529471 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-752be8-644dbd4459-4jzfq"] Oct 31 00:42:57.762825 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:57.762793 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1235cae5-9e24-46ef-879a-4c88ff2220cb" path="/var/lib/kubelet/pods/1235cae5-9e24-46ef-879a-4c88ff2220cb/volumes" Oct 31 00:42:58.222623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.222544 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f"] Oct 31 00:42:58.223412 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.223388 4969 cpu_manager.go:414] "containerMap: removing container" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" Oct 31 00:42:58.223529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.223415 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" Oct 31 00:42:58.223529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.223434 4969 cpu_manager.go:414] "containerMap: removing container" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" Oct 31 00:42:58.223529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.223442 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" Oct 31 00:42:58.223529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.223482 4969 cpu_manager.go:414] "containerMap: removing container" podUID="1235cae5-9e24-46ef-879a-4c88ff2220cb" containerName="maas-api" Oct 31 00:42:58.223529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.223489 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1235cae5-9e24-46ef-879a-4c88ff2220cb" containerName="maas-api" Oct 31 00:42:58.223840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.223661 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" Oct 31 00:42:58.223840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.223673 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f2becb-a214-43c1-8d2c-519a00bfd076" containerName="istio-proxy" Oct 31 00:42:58.223840 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.223690 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="1235cae5-9e24-46ef-879a-4c88ff2220cb" containerName="maas-api" Oct 31 00:42:58.229064 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.229040 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:42:58.282137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.282102 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-38dd3e\"" Oct 31 00:42:58.282574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.282247 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-38dd3e-dockercfg-kfl8q\"" Oct 31 00:42:58.283978 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.283953 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f"] Oct 31 00:42:58.306461 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.306432 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlqn\" (UniqueName: \"kubernetes.io/projected/468b1c7e-5913-4a04-807a-8ea0a8b28506-kube-api-access-zqlqn\") pod \"payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f\" (UID: \"468b1c7e-5913-4a04-807a-8ea0a8b28506\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:42:58.306657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.306567 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b1c7e-5913-4a04-807a-8ea0a8b28506-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f\" (UID: \"468b1c7e-5913-4a04-807a-8ea0a8b28506\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:42:58.374981 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.374950 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv"] Oct 31 00:42:58.379288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.379266 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:42:58.408042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.408010 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlqn\" (UniqueName: \"kubernetes.io/projected/468b1c7e-5913-4a04-807a-8ea0a8b28506-kube-api-access-zqlqn\") pod \"payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f\" (UID: \"468b1c7e-5913-4a04-807a-8ea0a8b28506\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:42:58.408220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.408086 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b1c7e-5913-4a04-807a-8ea0a8b28506-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f\" (UID: \"468b1c7e-5913-4a04-807a-8ea0a8b28506\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:42:58.408913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.408880 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b1c7e-5913-4a04-807a-8ea0a8b28506-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f\" (UID: \"468b1c7e-5913-4a04-807a-8ea0a8b28506\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:42:58.459036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.459001 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv"] Oct 31 00:42:58.472864 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.472756 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlqn\" (UniqueName: \"kubernetes.io/projected/468b1c7e-5913-4a04-807a-8ea0a8b28506-kube-api-access-zqlqn\") pod \"payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f\" (UID: \"468b1c7e-5913-4a04-807a-8ea0a8b28506\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:42:58.509690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.509657 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4qb\" (UniqueName: \"kubernetes.io/projected/cd621d4c-6bc9-44bf-9af8-089078325917-kube-api-access-gb4qb\") pod \"payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv\" (UID: \"cd621d4c-6bc9-44bf-9af8-089078325917\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:42:58.509860 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.509701 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd621d4c-6bc9-44bf-9af8-089078325917-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv\" (UID: \"cd621d4c-6bc9-44bf-9af8-089078325917\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:42:58.547297 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.547268 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:42:58.611363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.610567 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4qb\" (UniqueName: \"kubernetes.io/projected/cd621d4c-6bc9-44bf-9af8-089078325917-kube-api-access-gb4qb\") pod \"payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv\" (UID: \"cd621d4c-6bc9-44bf-9af8-089078325917\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:42:58.611363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.610666 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd621d4c-6bc9-44bf-9af8-089078325917-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv\" (UID: \"cd621d4c-6bc9-44bf-9af8-089078325917\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:42:58.611844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.611531 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd621d4c-6bc9-44bf-9af8-089078325917-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv\" (UID: \"cd621d4c-6bc9-44bf-9af8-089078325917\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:42:58.673301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.673265 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4qb\" (UniqueName: \"kubernetes.io/projected/cd621d4c-6bc9-44bf-9af8-089078325917-kube-api-access-gb4qb\") pod \"payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv\" (UID: \"cd621d4c-6bc9-44bf-9af8-089078325917\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:42:58.697108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.697067 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:42:58.927732 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:58.927703 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f"] Oct 31 00:42:58.928067 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:42:58.928043 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468b1c7e_5913_4a04_807a_8ea0a8b28506.slice/crio-f1c595ec50bd3949461eec407a804c825a11da05b5ad137943fbaf005822b92c WatchSource:0}: Error finding container f1c595ec50bd3949461eec407a804c825a11da05b5ad137943fbaf005822b92c: Status 404 returned error can't find the container with id f1c595ec50bd3949461eec407a804c825a11da05b5ad137943fbaf005822b92c Oct 31 00:42:59.084439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:59.084413 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv"] Oct 31 00:42:59.085633 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:42:59.085606 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd621d4c_6bc9_44bf_9af8_089078325917.slice/crio-159a8cefe09671fda3a285bdcbea449fe9a0464a623e7fcc91b478c2be6e5f1c WatchSource:0}: Error finding container 159a8cefe09671fda3a285bdcbea449fe9a0464a623e7fcc91b478c2be6e5f1c: Status 404 returned error can't find the container with id 159a8cefe09671fda3a285bdcbea449fe9a0464a623e7fcc91b478c2be6e5f1c Oct 31 00:42:59.385236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:59.385198 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" event={"ID":"cd621d4c-6bc9-44bf-9af8-089078325917","Type":"ContainerStarted","Data":"4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d"} Oct 31 00:42:59.385725 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:59.385384 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:42:59.385725 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:59.385433 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" event={"ID":"cd621d4c-6bc9-44bf-9af8-089078325917","Type":"ContainerStarted","Data":"159a8cefe09671fda3a285bdcbea449fe9a0464a623e7fcc91b478c2be6e5f1c"} Oct 31 00:42:59.386993 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:59.386971 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" event={"ID":"468b1c7e-5913-4a04-807a-8ea0a8b28506","Type":"ContainerStarted","Data":"ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00"} Oct 31 00:42:59.387102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:59.387005 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" event={"ID":"468b1c7e-5913-4a04-807a-8ea0a8b28506","Type":"ContainerStarted","Data":"f1c595ec50bd3949461eec407a804c825a11da05b5ad137943fbaf005822b92c"} Oct 31 00:42:59.387157 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:59.387104 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:42:59.517123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:59.517077 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" podStartSLOduration=1.5170625960000002 podStartE2EDuration="1.517062596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:59.51452692 +0000 UTC m=+2411.989885335" watchObservedRunningTime="2026-10-31 00:42:59.517062596 +0000 UTC m=+2411.992421040" Oct 31 00:42:59.645173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:42:59.644965 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" podStartSLOduration=1.644945447 podStartE2EDuration="1.644945447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:59.636376195 +0000 UTC m=+2412.111734619" watchObservedRunningTime="2026-10-31 00:42:59.644945447 +0000 UTC m=+2412.120303870" Oct 31 00:43:02.728537 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:02.728487 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:43:10.397001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:10.396962 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:43:10.397501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:10.397072 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:43:14.187029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.186991 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw"] Oct 31 00:43:14.187428 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.187193 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" podUID="c11b9467-7924-4ac3-bdc7-f05586998b88" containerName="payload-processing" containerID="cri-o://223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e" gracePeriod=30 Oct 31 00:43:14.297102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.297073 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb"] Oct 31 00:43:14.297277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.297255 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" podUID="aab82b8a-cef9-49f2-a946-81604ca84724" containerName="payload-pre-processing" containerID="cri-o://44953ffe10b8e18d33a0f0ec10f092cc8111be23c038cfcc825cbd50c4a8494d" gracePeriod=30 Oct 31 00:43:14.487157 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.487126 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:43:14.499342 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.499313 4969 generic.go:356] "Generic (PLEG): container finished" podID="aab82b8a-cef9-49f2-a946-81604ca84724" containerID="44953ffe10b8e18d33a0f0ec10f092cc8111be23c038cfcc825cbd50c4a8494d" exitCode=0 Oct 31 00:43:14.499501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.499386 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" event={"ID":"aab82b8a-cef9-49f2-a946-81604ca84724","Type":"ContainerDied","Data":"44953ffe10b8e18d33a0f0ec10f092cc8111be23c038cfcc825cbd50c4a8494d"} Oct 31 00:43:14.501284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.501249 4969 generic.go:356] "Generic (PLEG): container finished" podID="c11b9467-7924-4ac3-bdc7-f05586998b88" containerID="223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e" exitCode=0 Oct 31 00:43:14.501399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.501307 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" event={"ID":"c11b9467-7924-4ac3-bdc7-f05586998b88","Type":"ContainerDied","Data":"223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e"} Oct 31 00:43:14.501399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.501313 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" Oct 31 00:43:14.501399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.501350 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw" event={"ID":"c11b9467-7924-4ac3-bdc7-f05586998b88","Type":"ContainerDied","Data":"1f80582c59f9649259d3cf87551c7b1a9d5f9bbed7709769be4e70d42e476973"} Oct 31 00:43:14.501399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.501378 4969 scope.go:122] "RemoveContainer" containerID="223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e" Oct 31 00:43:14.512681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.512660 4969 scope.go:122] "RemoveContainer" containerID="223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e" Oct 31 00:43:14.513043 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:14.513017 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e\": container with ID starting with 223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e not found: ID does not exist" containerID="223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e" Oct 31 00:43:14.513167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.513054 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e"} err="failed to get container status \"223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e\": rpc error: code = NotFound desc = could not find container \"223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e\": container with ID starting with 223d3ca439350f7c55712b56e3d861458b0a74b08f2b8ef7b47c318ae264be5e not found: ID does not exist" Oct 31 00:43:14.513450 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.513427 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11b9467-7924-4ac3-bdc7-f05586998b88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11b9467-7924-4ac3-bdc7-f05586998b88-plugins-config-volume\") pod \"c11b9467-7924-4ac3-bdc7-f05586998b88\" (UID: \"c11b9467-7924-4ac3-bdc7-f05586998b88\") " Oct 31 00:43:14.513568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.513545 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11b9467-7924-4ac3-bdc7-f05586998b88-kube-api-access-jjnzc\" (UniqueName: \"kubernetes.io/projected/c11b9467-7924-4ac3-bdc7-f05586998b88-kube-api-access-jjnzc\") pod \"c11b9467-7924-4ac3-bdc7-f05586998b88\" (UID: \"c11b9467-7924-4ac3-bdc7-f05586998b88\") " Oct 31 00:43:14.514000 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.513825 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11b9467-7924-4ac3-bdc7-f05586998b88-plugins-config-volume" pod "c11b9467-7924-4ac3-bdc7-f05586998b88" (UID: "c11b9467-7924-4ac3-bdc7-f05586998b88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:14.514000 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.513987 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11b9467-7924-4ac3-bdc7-f05586998b88-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:14.516352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.516328 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11b9467-7924-4ac3-bdc7-f05586998b88-kube-api-access-jjnzc" pod "c11b9467-7924-4ac3-bdc7-f05586998b88" (UID: "c11b9467-7924-4ac3-bdc7-f05586998b88"). InnerVolumeSpecName "kube-api-access-jjnzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:14.576279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.576253 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:43:14.614513 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.614475 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab82b8a-cef9-49f2-a946-81604ca84724-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab82b8a-cef9-49f2-a946-81604ca84724-plugins-config-volume\") pod \"aab82b8a-cef9-49f2-a946-81604ca84724\" (UID: \"aab82b8a-cef9-49f2-a946-81604ca84724\") " Oct 31 00:43:14.614730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.614585 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab82b8a-cef9-49f2-a946-81604ca84724-kube-api-access-nrc4f\" (UniqueName: \"kubernetes.io/projected/aab82b8a-cef9-49f2-a946-81604ca84724-kube-api-access-nrc4f\") pod \"aab82b8a-cef9-49f2-a946-81604ca84724\" (UID: \"aab82b8a-cef9-49f2-a946-81604ca84724\") " Oct 31 00:43:14.614838 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.614818 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab82b8a-cef9-49f2-a946-81604ca84724-plugins-config-volume" pod "aab82b8a-cef9-49f2-a946-81604ca84724" (UID: "aab82b8a-cef9-49f2-a946-81604ca84724"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:14.615183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.615166 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnzc\" (UniqueName: \"kubernetes.io/projected/c11b9467-7924-4ac3-bdc7-f05586998b88-kube-api-access-jjnzc\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:14.615230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.615188 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab82b8a-cef9-49f2-a946-81604ca84724-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:14.617390 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.617360 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab82b8a-cef9-49f2-a946-81604ca84724-kube-api-access-nrc4f" pod "aab82b8a-cef9-49f2-a946-81604ca84724" (UID: "aab82b8a-cef9-49f2-a946-81604ca84724"). InnerVolumeSpecName "kube-api-access-nrc4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:14.716832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.716766 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrc4f\" (UniqueName: \"kubernetes.io/projected/aab82b8a-cef9-49f2-a946-81604ca84724-kube-api-access-nrc4f\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:14.937645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.937586 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw"] Oct 31 00:43:14.947164 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:14.947136 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6a8a5-5c49db7868-qdwzw"] Oct 31 00:43:15.029668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.029563 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp"] Oct 31 00:43:15.029824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.029780 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" podUID="271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" containerName="maas-api" containerID="cri-o://4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68" gracePeriod=30 Oct 31 00:43:15.321856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.321832 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:43:15.511027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.510985 4969 generic.go:356] "Generic (PLEG): container finished" podID="271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" containerID="4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68" exitCode=0 Oct 31 00:43:15.511192 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.511052 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" Oct 31 00:43:15.511192 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.511063 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" event={"ID":"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0","Type":"ContainerDied","Data":"4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68"} Oct 31 00:43:15.511192 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.511140 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp" event={"ID":"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0","Type":"ContainerDied","Data":"671d0244a6fa27dcd78b271916fb12ec24a4dfecb8117c320d6111afccb6f38a"} Oct 31 00:43:15.511192 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.511162 4969 scope.go:122] "RemoveContainer" containerID="4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68" Oct 31 00:43:15.512793 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.512772 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" Oct 31 00:43:15.513002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.512788 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb" event={"ID":"aab82b8a-cef9-49f2-a946-81604ca84724","Type":"ContainerDied","Data":"da2c74d05ef0e1dd28b796f3aa5724694479739bf1cb06015979380ba5e0361d"} Oct 31 00:43:15.523783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.523768 4969 scope.go:122] "RemoveContainer" containerID="4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68" Oct 31 00:43:15.524040 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:15.524013 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68\": container with ID starting with 4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68 not found: ID does not exist" containerID="4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68" Oct 31 00:43:15.524109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.524049 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68"} err="failed to get container status \"4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68\": rpc error: code = NotFound desc = could not find container \"4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68\": container with ID starting with 4662f2a931282710f1d636b46b00dccdf0d75952b19cb888ca2026f56979bb68 not found: ID does not exist" Oct 31 00:43:15.524109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.524066 4969 scope.go:122] "RemoveContainer" containerID="44953ffe10b8e18d33a0f0ec10f092cc8111be23c038cfcc825cbd50c4a8494d" Oct 31 00:43:15.526403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.526377 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls\") pod \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " Oct 31 00:43:15.526503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.526466 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-kube-api-access-m8wm6\" (UniqueName: \"kubernetes.io/projected/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-kube-api-access-m8wm6\") pod \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " Oct 31 00:43:15.526573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.526560 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-metrics-certs\") pod \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " Oct 31 00:43:15.526705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.526670 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-openshift-service-ca\") pod \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\" (UID: \"271b11b2-1a87-4c63-aa4b-8d2f56b28fc0\") " Oct 31 00:43:15.527279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.527201 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-openshift-service-ca" pod "271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" (UID: "271b11b2-1a87-4c63-aa4b-8d2f56b28fc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:15.529097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.529067 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls" pod "271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" (UID: "271b11b2-1a87-4c63-aa4b-8d2f56b28fc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:15.529184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.529169 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-kube-api-access-m8wm6" pod "271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" (UID: "271b11b2-1a87-4c63-aa4b-8d2f56b28fc0"). InnerVolumeSpecName "kube-api-access-m8wm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:15.529665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.529644 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-metrics-certs" pod "271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" (UID: "271b11b2-1a87-4c63-aa4b-8d2f56b28fc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:15.628115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.628083 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8wm6\" (UniqueName: \"kubernetes.io/projected/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-kube-api-access-m8wm6\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:15.628115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.628111 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:15.628115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.628123 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:15.628362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.628132 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:15.692907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.692872 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb"] Oct 31 00:43:15.714502 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.714475 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6a8a5-9bfd69b6c-fx2cb"] Oct 31 00:43:15.740882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.740829 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab82b8a-cef9-49f2-a946-81604ca84724" path="/var/lib/kubelet/pods/aab82b8a-cef9-49f2-a946-81604ca84724/volumes" Oct 31 00:43:15.741454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.741433 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11b9467-7924-4ac3-bdc7-f05586998b88" path="/var/lib/kubelet/pods/c11b9467-7924-4ac3-bdc7-f05586998b88/volumes" Oct 31 00:43:15.939466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.939383 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp"] Oct 31 00:43:15.943547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:15.943525 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2c02a9-67cf8d559-86nvp"] Oct 31 00:43:17.736875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:17.736837 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" path="/var/lib/kubelet/pods/271b11b2-1a87-4c63-aa4b-8d2f56b28fc0/volumes" Oct 31 00:43:19.015160 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:19.015131 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd"] Oct 31 00:43:22.905271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:22.905240 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4"] Oct 31 00:43:22.905711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:22.905446 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" containerID="cri-o://5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad" gracePeriod=30 Oct 31 00:43:24.015671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.015626 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" podUID="4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e" containerName="payload-pre-processing" containerID="cri-o://313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354" gracePeriod=25 Oct 31 00:43:24.290075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.290050 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:43:24.327366 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.327335 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-plugins-config-volume\") pod \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\" (UID: \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\") " Oct 31 00:43:24.327527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.327396 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-kube-api-access-4gg68\" (UniqueName: \"kubernetes.io/projected/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-kube-api-access-4gg68\") pod \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\" (UID: \"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e\") " Oct 31 00:43:24.327797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.327770 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-plugins-config-volume" pod "4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e" (UID: "4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:24.327974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.327956 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:24.329808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.329784 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-kube-api-access-4gg68" pod "4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e" (UID: "4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e"). InnerVolumeSpecName "kube-api-access-4gg68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:24.428903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.428869 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gg68\" (UniqueName: \"kubernetes.io/projected/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e-kube-api-access-4gg68\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:24.633302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.633270 4969 generic.go:356] "Generic (PLEG): container finished" podID="4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e" containerID="313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354" exitCode=0 Oct 31 00:43:24.633473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.633322 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" Oct 31 00:43:24.633473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.633355 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" event={"ID":"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e","Type":"ContainerDied","Data":"313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354"} Oct 31 00:43:24.633473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.633411 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd" event={"ID":"4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e","Type":"ContainerDied","Data":"3c26df5d5f33208577c1ac483fdbe7e5a53d8e81de1a0d49e67ad4ee74f3b90f"} Oct 31 00:43:24.633473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.633438 4969 scope.go:122] "RemoveContainer" containerID="313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354" Oct 31 00:43:24.645246 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.645230 4969 scope.go:122] "RemoveContainer" containerID="313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354" Oct 31 00:43:24.645512 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:24.645490 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354\": container with ID starting with 313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354 not found: ID does not exist" containerID="313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354" Oct 31 00:43:24.645698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.645518 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354"} err="failed to get container status \"313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354\": rpc error: code = NotFound desc = could not find container \"313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354\": container with ID starting with 313bc0b2475a74c9e4698c07a631457f17c17cbbe341e3bea8227ced9c1e6354 not found: ID does not exist" Oct 31 00:43:24.814079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.814036 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd"] Oct 31 00:43:24.818775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:24.818749 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-726d5b-6477c9868c-sv2rd"] Oct 31 00:43:25.735496 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:25.735458 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e" path="/var/lib/kubelet/pods/4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e/volumes" Oct 31 00:43:25.760862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:25.760825 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd"] Oct 31 00:43:25.761063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:25.761036 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" containerID="cri-o://144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4" gracePeriod=30 Oct 31 00:43:27.924861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.924556 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd"] Oct 31 00:43:27.925259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925169 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e" containerName="payload-pre-processing" Oct 31 00:43:27.925259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925183 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e" containerName="payload-pre-processing" Oct 31 00:43:27.925259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925221 4969 cpu_manager.go:414] "containerMap: removing container" podUID="aab82b8a-cef9-49f2-a946-81604ca84724" containerName="payload-pre-processing" Oct 31 00:43:27.925259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925226 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab82b8a-cef9-49f2-a946-81604ca84724" containerName="payload-pre-processing" Oct 31 00:43:27.925259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925241 4969 cpu_manager.go:414] "containerMap: removing container" podUID="271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" containerName="maas-api" Oct 31 00:43:27.925259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925246 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" containerName="maas-api" Oct 31 00:43:27.925259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925254 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c11b9467-7924-4ac3-bdc7-f05586998b88" containerName="payload-processing" Oct 31 00:43:27.925259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925259 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11b9467-7924-4ac3-bdc7-f05586998b88" containerName="payload-processing" Oct 31 00:43:27.925556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925343 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab82b8a-cef9-49f2-a946-81604ca84724" containerName="payload-pre-processing" Oct 31 00:43:27.925556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925352 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="271b11b2-1a87-4c63-aa4b-8d2f56b28fc0" containerName="maas-api" Oct 31 00:43:27.925556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925357 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11b9467-7924-4ac3-bdc7-f05586998b88" containerName="payload-processing" Oct 31 00:43:27.925556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.925365 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6bf62d-4d3b-4e98-b70f-0a211a7b3f0e" containerName="payload-pre-processing" Oct 31 00:43:27.928727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.928711 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.970315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.970272 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-socket\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.970610 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.970567 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-credential-socket\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.970765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.970748 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-istio-token\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.970877 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.970862 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-data\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.971001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.970986 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-envoy\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.971176 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.971160 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a960f229-1a93-4df8-afd8-025576f0e464-istiod-ca-cert\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.971284 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.971270 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a960f229-1a93-4df8-afd8-025576f0e464-istio-podinfo\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.971436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.971422 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-certs\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.971552 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.971539 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkhsz\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-kube-api-access-rkhsz\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:27.982971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:27.982925 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-7ae676-openshift-default-dockercfg-qmxgw\"" Oct 31 00:43:28.035417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.035391 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd"] Oct 31 00:43:28.073120 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073074 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-certs\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073120 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073118 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkhsz\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-kube-api-access-rkhsz\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073184 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-socket\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073222 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-credential-socket\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073255 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-istio-token\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073287 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-data\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073326 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-envoy\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073416 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a960f229-1a93-4df8-afd8-025576f0e464-istiod-ca-cert\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073443 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a960f229-1a93-4df8-afd8-025576f0e464-istio-podinfo\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073529 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-certs\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.073718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.073660 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-credential-socket\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.074188 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.074161 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-data\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.074338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.074312 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a960f229-1a93-4df8-afd8-025576f0e464-istiod-ca-cert\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.074861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.074837 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-socket\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.078097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.078062 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-envoy\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.083946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.083919 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a960f229-1a93-4df8-afd8-025576f0e464-istio-podinfo\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.137789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.137752 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-istio-token\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.137938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.137882 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkhsz\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-kube-api-access-rkhsz\") pod \"e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.169089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.169034 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:43:28.275372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275340 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-credential-socket\") pod \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " Oct 31 00:43:28.275542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275393 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istiod-ca-cert\") pod \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " Oct 31 00:43:28.275542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275435 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-data\") pod \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " Oct 31 00:43:28.275542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275465 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-certs\") pod \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " Oct 31 00:43:28.275542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275493 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-kube-api-access-prrzn\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-kube-api-access-prrzn\") pod \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " Oct 31 00:43:28.275768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275548 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-envoy\") pod \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " Oct 31 00:43:28.275768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275586 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-token\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-token\") pod \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " Oct 31 00:43:28.275768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275749 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-podinfo\") pod \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " Oct 31 00:43:28.275909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275765 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-credential-socket" pod "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" (UID: "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:28.275909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275828 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istiod-ca-cert" pod "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" (UID: "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:28.276022 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275896 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-data" pod "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" (UID: "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:28.276022 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.275914 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-socket\") pod \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\" (UID: \"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa\") " Oct 31 00:43:28.276134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.276098 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-certs" pod "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" (UID: "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:28.276188 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.276148 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-socket" pod "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" (UID: "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:28.276527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.276505 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:28.276677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.276529 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:28.276677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.276543 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:28.276677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.276557 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:28.276677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.276569 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:28.278134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.278113 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-envoy" pod "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" (UID: "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:28.278472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.278447 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-token" pod "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" (UID: "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:28.278765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.278740 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-podinfo" pod "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" (UID: "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:28.278859 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.278746 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-kube-api-access-prrzn" pod "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" (UID: "efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa"). InnerVolumeSpecName "kube-api-access-prrzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:28.307975 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.307940 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:28.377637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.377583 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prrzn\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-kube-api-access-prrzn\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:28.377768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.377643 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:28.377768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.377658 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:28.377768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.377672 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:28.674350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.674317 4969 generic.go:356] "Generic (PLEG): container finished" podID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerID="5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad" exitCode=0 Oct 31 00:43:28.674521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.674377 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" Oct 31 00:43:28.674521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.674404 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" event={"ID":"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa","Type":"ContainerDied","Data":"5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad"} Oct 31 00:43:28.674521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.674493 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4" event={"ID":"efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa","Type":"ContainerDied","Data":"4bd7f4cee3ff30f602d56f62a6afabccb05cb26774267743399ee115dc640cfc"} Oct 31 00:43:28.674700 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.674522 4969 scope.go:122] "RemoveContainer" containerID="5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad" Oct 31 00:43:28.688661 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.688642 4969 scope.go:122] "RemoveContainer" containerID="9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3" Oct 31 00:43:28.698363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.698342 4969 scope.go:122] "RemoveContainer" containerID="5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad" Oct 31 00:43:28.698636 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:28.698610 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad\": container with ID starting with 5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad not found: ID does not exist" containerID="5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad" Oct 31 00:43:28.698705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.698639 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad"} err="failed to get container status \"5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad\": rpc error: code = NotFound desc = could not find container \"5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad\": container with ID starting with 5581dc11d96927c94e2a3db75a95b5e75b74ff6d2dd65fb11f277cce31f337ad not found: ID does not exist" Oct 31 00:43:28.698705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.698657 4969 scope.go:122] "RemoveContainer" containerID="9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3" Oct 31 00:43:28.698969 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:28.698945 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3\": container with ID starting with 9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3 not found: ID does not exist" containerID="9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3" Oct 31 00:43:28.699054 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.698972 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3"} err="failed to get container status \"9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3\": rpc error: code = NotFound desc = could not find container \"9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3\": container with ID starting with 9e3c4b4fe4030395ae34cff33a15b784be2f43c8b859f799b11df4899239b1a3 not found: ID does not exist" Oct 31 00:43:28.757246 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.757219 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd"] Oct 31 00:43:28.781610 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:43:28.781566 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda960f229_1a93_4df8_afd8_025576f0e464.slice/crio-bb2a9495e2faebf98473a9b8a5fcd02b836905a8f6159bd81dca23103752286d WatchSource:0}: Error finding container bb2a9495e2faebf98473a9b8a5fcd02b836905a8f6159bd81dca23103752286d: Status 404 returned error can't find the container with id bb2a9495e2faebf98473a9b8a5fcd02b836905a8f6159bd81dca23103752286d Oct 31 00:43:28.783834 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.783806 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:43:28.783918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.783865 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:43:28.783918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.783895 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:43:28.850662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.850638 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4"] Oct 31 00:43:28.873070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:28.873040 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6a8a5-openshift-default-7584b4d997-jhjq4"] Oct 31 00:43:29.684647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:29.684582 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" event={"ID":"a960f229-1a93-4df8-afd8-025576f0e464","Type":"ContainerStarted","Data":"bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2"} Oct 31 00:43:29.685023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:29.684667 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" event={"ID":"a960f229-1a93-4df8-afd8-025576f0e464","Type":"ContainerStarted","Data":"bb2a9495e2faebf98473a9b8a5fcd02b836905a8f6159bd81dca23103752286d"} Oct 31 00:43:29.735098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:29.735067 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" path="/var/lib/kubelet/pods/efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa/volumes" Oct 31 00:43:29.818671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:29.818618 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" podStartSLOduration=2.818602122 podStartE2EDuration="2.818602122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:29.81524525 +0000 UTC m=+2442.290603672" watchObservedRunningTime="2026-10-31 00:43:29.818602122 +0000 UTC m=+2442.293960539" Oct 31 00:43:30.308264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:30.308232 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:30.314661 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:30.314630 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:30.692197 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:30.692168 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:30.693055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:30.693038 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:43:31.042572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.042548 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:43:31.108830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.108801 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-envoy\") pod \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " Oct 31 00:43:31.108971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.108868 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-podinfo\") pod \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " Oct 31 00:43:31.108971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.108941 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-socket\") pod \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " Oct 31 00:43:31.109072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.108972 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-kube-api-access-dcwgb\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-kube-api-access-dcwgb\") pod \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " Oct 31 00:43:31.109072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109008 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-data\") pod \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " Oct 31 00:43:31.109072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109066 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-token\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-token\") pod \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " Oct 31 00:43:31.109201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109115 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-certs\") pod \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " Oct 31 00:43:31.109201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109151 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istiod-ca-cert\") pod \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " Oct 31 00:43:31.109320 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109229 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-credential-socket\") pod \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\" (UID: \"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0\") " Oct 31 00:43:31.109320 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109258 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-socket" pod "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" (UID: "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:31.109320 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109306 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-data" pod "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" (UID: "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:31.109481 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109460 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-credential-socket" pod "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" (UID: "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:31.109556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109497 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-certs" pod "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" (UID: "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:31.109795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.109770 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istiod-ca-cert" pod "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" (UID: "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:31.110059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.110039 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.110122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.110061 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.110122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.110080 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.110122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.110097 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.110122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.110114 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.111417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.111391 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-kube-api-access-dcwgb" pod "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" (UID: "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0"). InnerVolumeSpecName "kube-api-access-dcwgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:31.111532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.111497 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-token" pod "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" (UID: "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:31.111792 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.111775 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-podinfo" pod "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" (UID: "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:31.111861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.111801 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-envoy" pod "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" (UID: "d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:31.211711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.211678 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcwgb\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-kube-api-access-dcwgb\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.211711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.211707 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.211711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.211717 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.211924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.211725 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:31.701671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.701634 4969 generic.go:356] "Generic (PLEG): container finished" podID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerID="144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4" exitCode=0 Oct 31 00:43:31.702076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.701694 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" Oct 31 00:43:31.702076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.701718 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" event={"ID":"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0","Type":"ContainerDied","Data":"144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4"} Oct 31 00:43:31.702076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.701786 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" event={"ID":"d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0","Type":"ContainerDied","Data":"74b96a984e2cfbdce52385549f4d310a7d1a53ee00b1ad74c0b19c05bbcdb9b4"} Oct 31 00:43:31.702076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.701816 4969 scope.go:122] "RemoveContainer" containerID="144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4" Oct 31 00:43:31.714176 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.714151 4969 scope.go:122] "RemoveContainer" containerID="cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604" Oct 31 00:43:31.724251 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.724229 4969 scope.go:122] "RemoveContainer" containerID="144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4" Oct 31 00:43:31.724526 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:31.724506 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4\": container with ID starting with 144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4 not found: ID does not exist" containerID="144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4" Oct 31 00:43:31.724616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.724537 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4"} err="failed to get container status \"144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4\": rpc error: code = NotFound desc = could not find container \"144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4\": container with ID starting with 144f238106e4bd1093f90f025bbe13bb8c72152f7c5f757b2ef11810ac82dca4 not found: ID does not exist" Oct 31 00:43:31.724616 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.724553 4969 scope.go:122] "RemoveContainer" containerID="cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604" Oct 31 00:43:31.724855 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:31.724826 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604\": container with ID starting with cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604 not found: ID does not exist" containerID="cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604" Oct 31 00:43:31.724901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.724864 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604"} err="failed to get container status \"cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604\": rpc error: code = NotFound desc = could not find container \"cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604\": container with ID starting with cb91d65000f1f75049dc034a1e75fd39401e1f6194282e4b91dc2608ec690604 not found: ID does not exist" Oct 31 00:43:31.876510 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.876481 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd"] Oct 31 00:43:31.926316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.926287 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd"] Oct 31 00:43:31.997644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.997571 4969 patch_prober.go:29] interesting pod/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:43:31.997785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:31.997650 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2c02a9-openshift-default-58744dbc8c-tqxzd" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:43:32.731991 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:32.731947 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:43:33.734715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:33.734678 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" path="/var/lib/kubelet/pods/d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0/volumes" Oct 31 00:43:36.495878 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.495840 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d"] Oct 31 00:43:36.496448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496434 4969 cpu_manager.go:414] "containerMap: removing container" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" Oct 31 00:43:36.496511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496450 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" Oct 31 00:43:36.496511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496460 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" Oct 31 00:43:36.496511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496465 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" Oct 31 00:43:36.496511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496472 4969 cpu_manager.go:414] "containerMap: removing container" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" Oct 31 00:43:36.496511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496478 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" Oct 31 00:43:36.496698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496527 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" Oct 31 00:43:36.496698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496532 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" Oct 31 00:43:36.496698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496640 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" Oct 31 00:43:36.496698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496651 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" Oct 31 00:43:36.496698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496658 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30e3b7f-f8d0-4d37-966a-0d3f6e0820b0" containerName="istio-proxy" Oct 31 00:43:36.496698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.496663 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbf9386-7d9d-4f0e-bbbd-da2a080cfdaa" containerName="istio-proxy" Oct 31 00:43:36.501775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.501757 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:36.559803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.559772 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d"] Oct 31 00:43:36.576466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.576430 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a110951-261d-44ac-9f18-c091b1a27591-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d\" (UID: \"4a110951-261d-44ac-9f18-c091b1a27591\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:36.576651 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.576482 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jb8\" (UniqueName: \"kubernetes.io/projected/4a110951-261d-44ac-9f18-c091b1a27591-kube-api-access-p6jb8\") pod \"payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d\" (UID: \"4a110951-261d-44ac-9f18-c091b1a27591\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:36.604100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.604071 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-7ae676-dockercfg-qk9qr\"" Oct 31 00:43:36.605002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.604977 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-7ae676\"" Oct 31 00:43:36.646031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.645996 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv"] Oct 31 00:43:36.650228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.650202 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:36.677740 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.677705 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a110951-261d-44ac-9f18-c091b1a27591-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d\" (UID: \"4a110951-261d-44ac-9f18-c091b1a27591\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:36.677873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.677745 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jb8\" (UniqueName: \"kubernetes.io/projected/4a110951-261d-44ac-9f18-c091b1a27591-kube-api-access-p6jb8\") pod \"payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d\" (UID: \"4a110951-261d-44ac-9f18-c091b1a27591\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:36.678338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.678319 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a110951-261d-44ac-9f18-c091b1a27591-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d\" (UID: \"4a110951-261d-44ac-9f18-c091b1a27591\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:36.708454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.708420 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv"] Oct 31 00:43:36.739670 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.739641 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jb8\" (UniqueName: \"kubernetes.io/projected/4a110951-261d-44ac-9f18-c091b1a27591-kube-api-access-p6jb8\") pod \"payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d\" (UID: \"4a110951-261d-44ac-9f18-c091b1a27591\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:36.779288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.779203 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxsv\" (UniqueName: \"kubernetes.io/projected/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-kube-api-access-frxsv\") pod \"payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv\" (UID: \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:36.779288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.779279 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv\" (UID: \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:36.815122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.815094 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:36.880416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.880345 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv\" (UID: \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:36.882220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.880566 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxsv\" (UniqueName: \"kubernetes.io/projected/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-kube-api-access-frxsv\") pod \"payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv\" (UID: \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:36.882220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.882017 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv\" (UID: \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:36.944253 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.944215 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxsv\" (UniqueName: \"kubernetes.io/projected/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-kube-api-access-frxsv\") pod \"payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv\" (UID: \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:36.965310 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:36.965273 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:37.211650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.209030 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d"] Oct 31 00:43:37.354960 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:43:37.354929 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bb5d6b_6d38_4422_bda2_31cdeee6339f.slice/crio-a19e495bbc0b53bcb78a8bc12ee4cf05ce894db6dafb9cd91b1c14f25fc8817a WatchSource:0}: Error finding container a19e495bbc0b53bcb78a8bc12ee4cf05ce894db6dafb9cd91b1c14f25fc8817a: Status 404 returned error can't find the container with id a19e495bbc0b53bcb78a8bc12ee4cf05ce894db6dafb9cd91b1c14f25fc8817a Oct 31 00:43:37.355218 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.355196 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv"] Oct 31 00:43:37.752391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.752355 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" event={"ID":"4a110951-261d-44ac-9f18-c091b1a27591","Type":"ContainerStarted","Data":"21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6"} Oct 31 00:43:37.752891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.752413 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" event={"ID":"4a110951-261d-44ac-9f18-c091b1a27591","Type":"ContainerStarted","Data":"9a8ec9a19432816f9ffd6589ca4d47b8325f122590a43fe25d0eafdde0650251"} Oct 31 00:43:37.752891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.752447 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:37.754663 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.754633 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" event={"ID":"a9bb5d6b-6d38-4422-bda2-31cdeee6339f","Type":"ContainerStarted","Data":"e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d"} Oct 31 00:43:37.754789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.754689 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" event={"ID":"a9bb5d6b-6d38-4422-bda2-31cdeee6339f","Type":"ContainerStarted","Data":"a19e495bbc0b53bcb78a8bc12ee4cf05ce894db6dafb9cd91b1c14f25fc8817a"} Oct 31 00:43:37.754789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.754718 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:37.853025 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.852985 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr"] Oct 31 00:43:37.853283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.853254 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" podUID="69df93e1-3356-4017-b8fb-6429c2383bab" containerName="maas-api" containerID="cri-o://f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c" gracePeriod=30 Oct 31 00:43:37.871891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.871848 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" podStartSLOduration=1.871835483 podStartE2EDuration="1.871835483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:37.868178418 +0000 UTC m=+2450.343536842" watchObservedRunningTime="2026-10-31 00:43:37.871835483 +0000 UTC m=+2450.347193905" Oct 31 00:43:37.986105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:37.986060 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" podStartSLOduration=1.986042845 podStartE2EDuration="1.986042845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:37.984713482 +0000 UTC m=+2450.460071907" watchObservedRunningTime="2026-10-31 00:43:37.986042845 +0000 UTC m=+2450.461401268" Oct 31 00:43:38.132295 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.132266 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:43:38.195434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.195399 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69df93e1-3356-4017-b8fb-6429c2383bab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69df93e1-3356-4017-b8fb-6429c2383bab-openshift-service-ca\") pod \"69df93e1-3356-4017-b8fb-6429c2383bab\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " Oct 31 00:43:38.195624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.195470 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls\") pod \"69df93e1-3356-4017-b8fb-6429c2383bab\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " Oct 31 00:43:38.195624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.195528 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-metrics-certs\") pod \"69df93e1-3356-4017-b8fb-6429c2383bab\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " Oct 31 00:43:38.195733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.195649 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69df93e1-3356-4017-b8fb-6429c2383bab-kube-api-access-65l4c\" (UniqueName: \"kubernetes.io/projected/69df93e1-3356-4017-b8fb-6429c2383bab-kube-api-access-65l4c\") pod \"69df93e1-3356-4017-b8fb-6429c2383bab\" (UID: \"69df93e1-3356-4017-b8fb-6429c2383bab\") " Oct 31 00:43:38.195883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.195861 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69df93e1-3356-4017-b8fb-6429c2383bab-openshift-service-ca" pod "69df93e1-3356-4017-b8fb-6429c2383bab" (UID: "69df93e1-3356-4017-b8fb-6429c2383bab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:38.196117 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.196094 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69df93e1-3356-4017-b8fb-6429c2383bab-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:38.197880 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.197851 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-metrics-certs" pod "69df93e1-3356-4017-b8fb-6429c2383bab" (UID: "69df93e1-3356-4017-b8fb-6429c2383bab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:38.197970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.197917 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69df93e1-3356-4017-b8fb-6429c2383bab-kube-api-access-65l4c" pod "69df93e1-3356-4017-b8fb-6429c2383bab" (UID: "69df93e1-3356-4017-b8fb-6429c2383bab"). InnerVolumeSpecName "kube-api-access-65l4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:38.197970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.197948 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls" pod "69df93e1-3356-4017-b8fb-6429c2383bab" (UID: "69df93e1-3356-4017-b8fb-6429c2383bab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:38.297071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.297000 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:38.297071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.297026 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69df93e1-3356-4017-b8fb-6429c2383bab-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:38.297071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.297039 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65l4c\" (UniqueName: \"kubernetes.io/projected/69df93e1-3356-4017-b8fb-6429c2383bab-kube-api-access-65l4c\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:38.415784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.415751 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss"] Oct 31 00:43:38.415968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.415947 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" podUID="2318af60-6336-43f6-a954-33f0cc497337" containerName="payload-pre-processing" containerID="cri-o://db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba" gracePeriod=30 Oct 31 00:43:38.703721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.703326 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:43:38.763162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.763131 4969 generic.go:356] "Generic (PLEG): container finished" podID="2318af60-6336-43f6-a954-33f0cc497337" containerID="db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba" exitCode=0 Oct 31 00:43:38.763585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.763169 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" Oct 31 00:43:38.763585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.763219 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" event={"ID":"2318af60-6336-43f6-a954-33f0cc497337","Type":"ContainerDied","Data":"db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba"} Oct 31 00:43:38.763585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.763272 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss" event={"ID":"2318af60-6336-43f6-a954-33f0cc497337","Type":"ContainerDied","Data":"390ce14463d63d941a742eadf33b0be2ba6934968a0b54df2f87544a5e220913"} Oct 31 00:43:38.763585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.763327 4969 scope.go:122] "RemoveContainer" containerID="db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba" Oct 31 00:43:38.764849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.764818 4969 generic.go:356] "Generic (PLEG): container finished" podID="69df93e1-3356-4017-b8fb-6429c2383bab" containerID="f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c" exitCode=0 Oct 31 00:43:38.765118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.764852 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" event={"ID":"69df93e1-3356-4017-b8fb-6429c2383bab","Type":"ContainerDied","Data":"f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c"} Oct 31 00:43:38.765118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.764877 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" Oct 31 00:43:38.765118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.764900 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr" event={"ID":"69df93e1-3356-4017-b8fb-6429c2383bab","Type":"ContainerDied","Data":"801ce049356133d11d5e7b91b41fe7154f4ef67c1fc89ff7e79b775759702dcf"} Oct 31 00:43:38.774163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.774147 4969 scope.go:122] "RemoveContainer" containerID="db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba" Oct 31 00:43:38.774439 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:38.774420 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba\": container with ID starting with db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba not found: ID does not exist" containerID="db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba" Oct 31 00:43:38.774524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.774451 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba"} err="failed to get container status \"db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba\": rpc error: code = NotFound desc = could not find container \"db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba\": container with ID starting with db54be773f1adc0aeb4275b48a9e957355a8e17c981338a5a1507ac83912fdba not found: ID does not exist" Oct 31 00:43:38.774524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.774474 4969 scope.go:122] "RemoveContainer" containerID="f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c" Oct 31 00:43:38.784656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.784637 4969 scope.go:122] "RemoveContainer" containerID="f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c" Oct 31 00:43:38.784930 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:38.784910 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c\": container with ID starting with f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c not found: ID does not exist" containerID="f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c" Oct 31 00:43:38.784988 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.784943 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c"} err="failed to get container status \"f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c\": rpc error: code = NotFound desc = could not find container \"f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c\": container with ID starting with f7e039e5d1a7a3c402039fdbb2f1234793f91c832e2c6f41038ece34c0998f8c not found: ID does not exist" Oct 31 00:43:38.803382 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.803358 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2318af60-6336-43f6-a954-33f0cc497337-kube-api-access-pvrqd\" (UniqueName: \"kubernetes.io/projected/2318af60-6336-43f6-a954-33f0cc497337-kube-api-access-pvrqd\") pod \"2318af60-6336-43f6-a954-33f0cc497337\" (UID: \"2318af60-6336-43f6-a954-33f0cc497337\") " Oct 31 00:43:38.803560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.803548 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2318af60-6336-43f6-a954-33f0cc497337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2318af60-6336-43f6-a954-33f0cc497337-plugins-config-volume\") pod \"2318af60-6336-43f6-a954-33f0cc497337\" (UID: \"2318af60-6336-43f6-a954-33f0cc497337\") " Oct 31 00:43:38.803951 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.803928 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2318af60-6336-43f6-a954-33f0cc497337-plugins-config-volume" pod "2318af60-6336-43f6-a954-33f0cc497337" (UID: "2318af60-6336-43f6-a954-33f0cc497337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:38.804125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.804102 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2318af60-6336-43f6-a954-33f0cc497337-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:38.805669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.805648 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2318af60-6336-43f6-a954-33f0cc497337-kube-api-access-pvrqd" pod "2318af60-6336-43f6-a954-33f0cc497337" (UID: "2318af60-6336-43f6-a954-33f0cc497337"). InnerVolumeSpecName "kube-api-access-pvrqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:38.905173 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.905143 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvrqd\" (UniqueName: \"kubernetes.io/projected/2318af60-6336-43f6-a954-33f0cc497337-kube-api-access-pvrqd\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:43:38.993417 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.993385 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr"] Oct 31 00:43:38.997920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:38.997893 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1a4f30-6b5779767-znxsr"] Oct 31 00:43:39.188668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:39.188583 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss"] Oct 31 00:43:39.196745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:39.196723 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-689dbb59f9-cqhss"] Oct 31 00:43:39.735307 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:39.735273 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2318af60-6336-43f6-a954-33f0cc497337" path="/var/lib/kubelet/pods/2318af60-6336-43f6-a954-33f0cc497337/volumes" Oct 31 00:43:39.735759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:39.735746 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69df93e1-3356-4017-b8fb-6429c2383bab" path="/var/lib/kubelet/pods/69df93e1-3356-4017-b8fb-6429c2383bab/volumes" Oct 31 00:43:45.954675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:45.954620 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td"] Oct 31 00:43:45.955462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:45.955437 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2318af60-6336-43f6-a954-33f0cc497337" containerName="payload-pre-processing" Oct 31 00:43:45.955574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:45.955464 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2318af60-6336-43f6-a954-33f0cc497337" containerName="payload-pre-processing" Oct 31 00:43:45.955574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:45.955526 4969 cpu_manager.go:414] "containerMap: removing container" podUID="69df93e1-3356-4017-b8fb-6429c2383bab" containerName="maas-api" Oct 31 00:43:45.955574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:45.955534 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69df93e1-3356-4017-b8fb-6429c2383bab" containerName="maas-api" Oct 31 00:43:45.955784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:45.955708 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="69df93e1-3356-4017-b8fb-6429c2383bab" containerName="maas-api" Oct 31 00:43:45.955784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:45.955732 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2318af60-6336-43f6-a954-33f0cc497337" containerName="payload-pre-processing" Oct 31 00:43:45.959149 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:45.959128 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.064758 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.064717 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-87b0f8\"/\"default-dockercfg-fqp7q\"" Oct 31 00:43:46.064758 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.064752 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-87b0f8\"/\"openshift-service-ca.crt\"" Oct 31 00:43:46.064990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.064789 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-87b0f8\"/\"kube-root-ca.crt\"" Oct 31 00:43:46.064990 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.064937 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-87b0f8\"/\"praxis-route-87b0f8-kserve-self-signed-certs\"" Oct 31 00:43:46.068332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.068312 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td"] Oct 31 00:43:46.095876 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.095843 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-dshm\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.096049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.095983 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-tmp-dir\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.096049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.096035 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkf6s\" (UniqueName: \"kubernetes.io/projected/976ab5cb-78c2-4508-8066-37627fb32aa0-kube-api-access-fkf6s\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.096129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.096060 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-model-cache\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.096201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.096179 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-home\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.096243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.096221 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976ab5cb-78c2-4508-8066-37627fb32aa0-tls-certs\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.197538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.197495 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-dshm\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.197768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.197628 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-tmp-dir\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.197768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.197674 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkf6s\" (UniqueName: \"kubernetes.io/projected/976ab5cb-78c2-4508-8066-37627fb32aa0-kube-api-access-fkf6s\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.197768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.197699 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-model-cache\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.197768 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.197757 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-home\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.197974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.197808 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976ab5cb-78c2-4508-8066-37627fb32aa0-tls-certs\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.198067 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.198044 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-tmp-dir\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.198163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.198143 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-home\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.198237 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.198119 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-model-cache\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.200098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.200072 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-dshm\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.200542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.200521 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976ab5cb-78c2-4508-8066-37627fb32aa0-tls-certs\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.291509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.291434 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkf6s\" (UniqueName: \"kubernetes.io/projected/976ab5cb-78c2-4508-8066-37627fb32aa0-kube-api-access-fkf6s\") pod \"praxis-route-87b0f8-kserve-56b97cbf9-pt6td\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.573441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.573361 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:46.965286 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:46.965261 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td"] Oct 31 00:43:46.965677 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:43:46.965539 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod976ab5cb_78c2_4508_8066_37627fb32aa0.slice/crio-9286708004f42e28a9282f190da356cc9edf6db4a6c6fe3df42581389abcd04d WatchSource:0}: Error finding container 9286708004f42e28a9282f190da356cc9edf6db4a6c6fe3df42581389abcd04d: Status 404 returned error can't find the container with id 9286708004f42e28a9282f190da356cc9edf6db4a6c6fe3df42581389abcd04d Oct 31 00:43:47.834240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:47.834206 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" event={"ID":"976ab5cb-78c2-4508-8066-37627fb32aa0","Type":"ContainerStarted","Data":"47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f"} Oct 31 00:43:47.834410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:47.834253 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" event={"ID":"976ab5cb-78c2-4508-8066-37627fb32aa0","Type":"ContainerStarted","Data":"9286708004f42e28a9282f190da356cc9edf6db4a6c6fe3df42581389abcd04d"} Oct 31 00:43:47.960244 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:47.960182 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" podStartSLOduration=2.9601632159999998 podStartE2EDuration="2.960163216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:47.956877235 +0000 UTC m=+2460.432235652" watchObservedRunningTime="2026-10-31 00:43:47.960163216 +0000 UTC m=+2460.435521641" Oct 31 00:43:48.573739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:48.573703 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:48.767680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:48.767642 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:43:48.767858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:48.767712 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:43:50.526149 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.526118 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt"] Oct 31 00:43:50.531895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.531876 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.553352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.553324 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47310c95-58a8-4814-9668-f93cdc6231e6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.553494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.553372 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v76hg\" (UniqueName: \"kubernetes.io/projected/47310c95-58a8-4814-9668-f93cdc6231e6-kube-api-access-v76hg\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.553494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.553422 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.553631 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.553506 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-metrics-certs\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.584072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.584049 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-38dd3e-serving-cert\"" Oct 31 00:43:50.637376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.637346 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt"] Oct 31 00:43:50.654260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.654230 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.654453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.654271 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-metrics-certs\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.654453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.654343 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47310c95-58a8-4814-9668-f93cdc6231e6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.654453 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:50.654366 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-serving-cert: secret "maas-api-e2e-praxis-a-38dd3e-serving-cert" not found Oct 31 00:43:50.654453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.654382 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v76hg\" (UniqueName: \"kubernetes.io/projected/47310c95-58a8-4814-9668-f93cdc6231e6-kube-api-access-v76hg\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.654453 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:43:50.654422 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls podName:47310c95-58a8-4814-9668-f93cdc6231e6 nodeName:}" failed. No retries permitted until 2026-10-31 00:43:51.154409247 +0000 UTC m=+2463.629767682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls") pod "maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" (UID: "47310c95-58a8-4814-9668-f93cdc6231e6") : secret "maas-api-e2e-praxis-a-38dd3e-serving-cert" not found Oct 31 00:43:50.655121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.655102 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47310c95-58a8-4814-9668-f93cdc6231e6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.657194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.657159 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-metrics-certs\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.716645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.716618 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v76hg\" (UniqueName: \"kubernetes.io/projected/47310c95-58a8-4814-9668-f93cdc6231e6-kube-api-access-v76hg\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:50.842338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.842255 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk"] Oct 31 00:43:50.846456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.846440 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.856228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.856202 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-data\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.856355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.856247 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-token\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.856355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.856315 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-certs\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.856355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.856348 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-socket\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.856501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.856371 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-credential-socket\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.856501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.856391 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5wn\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-kube-api-access-8m5wn\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.856501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.856473 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-podinfo\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.856677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.856520 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-envoy\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.856733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.856669 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659e33-767b-45ce-98b6-3defaa5924c1-istiod-ca-cert\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.917823 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.917786 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-6a693b-openshift-default-dockercfg-7h4fw\"" Oct 31 00:43:50.923118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.922885 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk"] Oct 31 00:43:50.958384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958342 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-data\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.958566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958402 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-token\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.958566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958435 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-certs\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.958566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958473 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-socket\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.958566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958511 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-credential-socket\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.958566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958541 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5wn\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-kube-api-access-8m5wn\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.958927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958631 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-podinfo\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.958927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958701 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-envoy\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.958927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958763 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659e33-767b-45ce-98b6-3defaa5924c1-istiod-ca-cert\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.958927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.958837 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-data\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.959480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.959441 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-socket\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.959772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.959703 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-certs\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.959772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.959723 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-credential-socket\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.960104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.960076 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659e33-767b-45ce-98b6-3defaa5924c1-istiod-ca-cert\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.961528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.961506 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-podinfo\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:50.962298 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:50.962279 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-envoy\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:51.023189 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.023152 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5wn\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-kube-api-access-8m5wn\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:51.023385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.023292 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-token\") pod \"e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:51.160919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.160886 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:51.163186 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.163158 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:51.163414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.163392 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:51.445586 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.445518 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:51.599090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.599059 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk"] Oct 31 00:43:51.599705 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:43:51.599664 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e659e33_767b_45ce_98b6_3defaa5924c1.slice/crio-f18894ed73b7298092b486e5915719e4c84e146be0642aa450d2883b223e68f9 WatchSource:0}: Error finding container f18894ed73b7298092b486e5915719e4c84e146be0642aa450d2883b223e68f9: Status 404 returned error can't find the container with id f18894ed73b7298092b486e5915719e4c84e146be0642aa450d2883b223e68f9 Oct 31 00:43:51.602871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.602840 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:43:51.602956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.602903 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:43:51.602956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.602930 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:43:51.826419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.826393 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt"] Oct 31 00:43:51.826419 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:43:51.826404 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47310c95_58a8_4814_9668_f93cdc6231e6.slice/crio-d15b358829b7b4ff09da81c17f2e75716e54afb6f2e3cbc4612827754092026e WatchSource:0}: Error finding container d15b358829b7b4ff09da81c17f2e75716e54afb6f2e3cbc4612827754092026e: Status 404 returned error can't find the container with id d15b358829b7b4ff09da81c17f2e75716e54afb6f2e3cbc4612827754092026e Oct 31 00:43:51.827992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.827971 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:43:51.867792 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.867751 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" event={"ID":"5e659e33-767b-45ce-98b6-3defaa5924c1","Type":"ContainerStarted","Data":"3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4"} Oct 31 00:43:51.867980 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.867815 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" event={"ID":"5e659e33-767b-45ce-98b6-3defaa5924c1","Type":"ContainerStarted","Data":"f18894ed73b7298092b486e5915719e4c84e146be0642aa450d2883b223e68f9"} Oct 31 00:43:51.869151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.869124 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" event={"ID":"47310c95-58a8-4814-9668-f93cdc6231e6","Type":"ContainerStarted","Data":"d15b358829b7b4ff09da81c17f2e75716e54afb6f2e3cbc4612827754092026e"} Oct 31 00:43:51.993916 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:51.993862 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" podStartSLOduration=1.993847643 podStartE2EDuration="1.993847643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:51.992287567 +0000 UTC m=+2464.467645991" watchObservedRunningTime="2026-10-31 00:43:51.993847643 +0000 UTC m=+2464.469206065" Oct 31 00:43:52.163331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:52.163303 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:52.168612 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:52.168573 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:52.878149 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:52.878110 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" event={"ID":"47310c95-58a8-4814-9668-f93cdc6231e6","Type":"ContainerStarted","Data":"dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40"} Oct 31 00:43:52.878767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:52.878447 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:52.878767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:52.878496 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:43:52.879456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:52.879438 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:43:53.009547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:53.009435 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" podStartSLOduration=2.187702527 podStartE2EDuration="3.009420266s" totalImagesPullingTime="821.717739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:53.006160769 +0000 UTC m=+2465.481519194" watchObservedRunningTime="2026-10-31 00:43:53.009420266 +0000 UTC m=+2465.484778689" Oct 31 00:43:56.574133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:56.574098 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:56.586490 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:56.586465 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:56.916706 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:56.916680 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:43:58.890669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:43:58.890581 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:44:02.727738 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:44:02.727701 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:44:23.514146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:23.514108 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf"] Oct 31 00:44:23.514732 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:23.514334 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" containerID="cri-o://7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb" gracePeriod=30 Oct 31 00:44:26.066153 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:26.066116 4969 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:44:26.066553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:26.066174 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:28.842934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.842907 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:44:28.921422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921392 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-token\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-token\") pod \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " Oct 31 00:44:28.921619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921530 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-credential-socket\") pod \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " Oct 31 00:44:28.921619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921553 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istiod-ca-cert\") pod \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " Oct 31 00:44:28.921718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921618 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-socket\") pod \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " Oct 31 00:44:28.921718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921663 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-kube-api-access-6hzlb\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-kube-api-access-6hzlb\") pod \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " Oct 31 00:44:28.921718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921693 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-certs\") pod \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " Oct 31 00:44:28.921867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921788 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-data\") pod \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " Oct 31 00:44:28.921867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921852 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-podinfo\") pod \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " Oct 31 00:44:28.922004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921881 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-envoy\") pod \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\" (UID: \"edf6cadb-2c52-4481-a3a9-f20c6228f48a\") " Oct 31 00:44:28.922004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921870 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-credential-socket" pod "edf6cadb-2c52-4481-a3a9-f20c6228f48a" (UID: "edf6cadb-2c52-4481-a3a9-f20c6228f48a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:28.922004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921903 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-socket" pod "edf6cadb-2c52-4481-a3a9-f20c6228f48a" (UID: "edf6cadb-2c52-4481-a3a9-f20c6228f48a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:28.922004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.921962 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istiod-ca-cert" pod "edf6cadb-2c52-4481-a3a9-f20c6228f48a" (UID: "edf6cadb-2c52-4481-a3a9-f20c6228f48a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:28.922219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.922096 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-certs" pod "edf6cadb-2c52-4481-a3a9-f20c6228f48a" (UID: "edf6cadb-2c52-4481-a3a9-f20c6228f48a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:28.922219 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.922109 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-data" pod "edf6cadb-2c52-4481-a3a9-f20c6228f48a" (UID: "edf6cadb-2c52-4481-a3a9-f20c6228f48a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:28.922386 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.922363 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:28.922386 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.922383 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:28.922780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.922393 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:28.922780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.922403 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:28.922780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.922412 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:28.924348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.924324 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-podinfo" pod "edf6cadb-2c52-4481-a3a9-f20c6228f48a" (UID: "edf6cadb-2c52-4481-a3a9-f20c6228f48a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:44:28.924433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.924382 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-kube-api-access-6hzlb" pod "edf6cadb-2c52-4481-a3a9-f20c6228f48a" (UID: "edf6cadb-2c52-4481-a3a9-f20c6228f48a"). InnerVolumeSpecName "kube-api-access-6hzlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:28.924919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.924898 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-envoy" pod "edf6cadb-2c52-4481-a3a9-f20c6228f48a" (UID: "edf6cadb-2c52-4481-a3a9-f20c6228f48a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:28.925002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:28.924918 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-token" pod "edf6cadb-2c52-4481-a3a9-f20c6228f48a" (UID: "edf6cadb-2c52-4481-a3a9-f20c6228f48a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:29.023220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.023189 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:29.023220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.023218 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:29.023220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.023228 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:29.023458 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.023237 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzlb\" (UniqueName: \"kubernetes.io/projected/edf6cadb-2c52-4481-a3a9-f20c6228f48a-kube-api-access-6hzlb\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:29.130403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.130372 4969 generic.go:356] "Generic (PLEG): container finished" podID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerID="7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb" exitCode=0 Oct 31 00:44:29.130584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.130418 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" Oct 31 00:44:29.130584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.130448 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" event={"ID":"edf6cadb-2c52-4481-a3a9-f20c6228f48a","Type":"ContainerDied","Data":"7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb"} Oct 31 00:44:29.130584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.130499 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf" event={"ID":"edf6cadb-2c52-4481-a3a9-f20c6228f48a","Type":"ContainerDied","Data":"6a51776c049e736f31ab000c40e954875469b6c3e622f58464c844c2fc9bbe13"} Oct 31 00:44:29.130584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.130526 4969 scope.go:122] "RemoveContainer" containerID="7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb" Oct 31 00:44:29.142507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.142491 4969 scope.go:122] "RemoveContainer" containerID="7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb" Oct 31 00:44:29.142794 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:44:29.142775 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb\": container with ID starting with 7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb not found: ID does not exist" containerID="7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb" Oct 31 00:44:29.142863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.142802 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb"} err="failed to get container status \"7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb\": rpc error: code = NotFound desc = could not find container \"7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb\": container with ID starting with 7abfbf19595aeb92a640a4063c5e493d9cbc61c8df964b446bbabadc4f82cceb not found: ID does not exist" Oct 31 00:44:29.307763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.307725 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf"] Oct 31 00:44:29.318864 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.318837 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-9cvqf"] Oct 31 00:44:29.734886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:29.734851 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" path="/var/lib/kubelet/pods/edf6cadb-2c52-4481-a3a9-f20c6228f48a/volumes" Oct 31 00:44:32.728148 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:44:32.728110 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:44:36.181518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:36.181482 4969 generic.go:356] "Generic (PLEG): container finished" podID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerID="407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166" exitCode=0 Oct 31 00:44:36.181983 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:36.181558 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" event={"ID":"713e3d53-958f-4f7a-bdbb-645c326011e4","Type":"ContainerDied","Data":"407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166"} Oct 31 00:44:36.182028 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:36.182004 4969 scope.go:122] "RemoveContainer" containerID="407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166" Oct 31 00:44:36.183435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:36.183403 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:36.183546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:36.183487 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:36.183546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:36.183534 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:37.191531 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:37.191465 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" event={"ID":"713e3d53-958f-4f7a-bdbb-645c326011e4","Type":"ContainerStarted","Data":"aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc"} Oct 31 00:44:37.312046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:37.312005 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:44:38.013014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:38.012975 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:44:38.014493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:38.014464 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 31 00:44:38.014640 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:38.014514 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 31 00:44:39.012144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:39.012106 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 31 00:44:39.012624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:39.012160 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 31 00:44:40.013207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:40.013171 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 31 00:44:40.013639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:40.013261 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 31 00:44:41.012303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:41.012266 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 31 00:44:41.012493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:41.012325 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 31 00:44:42.013088 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:42.013045 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 31 00:44:42.013553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:42.013110 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 31 00:44:43.061253 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:43.061217 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:44:43.063034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:43.062985 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:44:44.321624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.321554 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6"] Oct 31 00:44:44.322358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.322335 4969 cpu_manager.go:414] "containerMap: removing container" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" Oct 31 00:44:44.322465 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.322359 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" Oct 31 00:44:44.322577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.322563 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf6cadb-2c52-4481-a3a9-f20c6228f48a" containerName="istio-proxy" Oct 31 00:44:44.328238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.326748 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.398923 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.398890 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-6a693b-serving-cert\"" Oct 31 00:44:44.409734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.409703 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6"] Oct 31 00:44:44.452137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.452101 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.452304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.452149 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-metrics-certs\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.452368 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.452325 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f649\" (UniqueName: \"kubernetes.io/projected/403b3bf9-5ad9-47fc-831e-d03951f9573a-kube-api-access-6f649\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.452459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.452442 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403b3bf9-5ad9-47fc-831e-d03951f9573a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.553166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.553129 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.553369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.553176 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-metrics-certs\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.553369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.553257 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f649\" (UniqueName: \"kubernetes.io/projected/403b3bf9-5ad9-47fc-831e-d03951f9573a-kube-api-access-6f649\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.553369 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:44:44.553298 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-serving-cert: secret "maas-api-e2e-praxis-a-6a693b-serving-cert" not found Oct 31 00:44:44.553369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.553310 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403b3bf9-5ad9-47fc-831e-d03951f9573a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.553583 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:44:44.553379 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls podName:403b3bf9-5ad9-47fc-831e-d03951f9573a nodeName:}" failed. No retries permitted until 2026-10-31 00:44:45.053359697 +0000 UTC m=+2517.528718099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls") pod "maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" (UID: "403b3bf9-5ad9-47fc-831e-d03951f9573a") : secret "maas-api-e2e-praxis-a-6a693b-serving-cert" not found Oct 31 00:44:44.554025 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.554005 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403b3bf9-5ad9-47fc-831e-d03951f9573a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.556006 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.555978 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-metrics-certs\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:44.622637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:44.622569 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f649\" (UniqueName: \"kubernetes.io/projected/403b3bf9-5ad9-47fc-831e-d03951f9573a-kube-api-access-6f649\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:45.059996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:45.059897 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:45.062494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:45.062470 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:45.247053 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:45.247020 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:45.654728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:45.654699 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6"] Oct 31 00:44:45.655979 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:44:45.655949 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403b3bf9_5ad9_47fc_831e_d03951f9573a.slice/crio-8080a292bbc2b3a7ce545918b1305a70b420e8d75685717f8927068347924815 WatchSource:0}: Error finding container 8080a292bbc2b3a7ce545918b1305a70b420e8d75685717f8927068347924815: Status 404 returned error can't find the container with id 8080a292bbc2b3a7ce545918b1305a70b420e8d75685717f8927068347924815 Oct 31 00:44:46.266819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:46.266785 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" event={"ID":"403b3bf9-5ad9-47fc-831e-d03951f9573a","Type":"ContainerStarted","Data":"8080a292bbc2b3a7ce545918b1305a70b420e8d75685717f8927068347924815"} Oct 31 00:44:47.276477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:47.276444 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" event={"ID":"403b3bf9-5ad9-47fc-831e-d03951f9573a","Type":"ContainerStarted","Data":"d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658"} Oct 31 00:44:47.277032 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:47.276524 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:47.402965 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:47.402907 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" podStartSLOduration=2.513476895 podStartE2EDuration="3.402892097s" totalImagesPullingTime="889.415202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:47.401932881 +0000 UTC m=+2519.877291306" watchObservedRunningTime="2026-10-31 00:44:47.402892097 +0000 UTC m=+2519.878250519" Oct 31 00:44:51.312076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:51.311999 4969 generic.go:356] "Generic (PLEG): container finished" podID="a960f229-1a93-4df8-afd8-025576f0e464" containerID="bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2" exitCode=0 Oct 31 00:44:51.312477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:51.312080 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" event={"ID":"a960f229-1a93-4df8-afd8-025576f0e464","Type":"ContainerDied","Data":"bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2"} Oct 31 00:44:51.312522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:51.312505 4969 scope.go:122] "RemoveContainer" containerID="bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2" Oct 31 00:44:51.314012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:51.313980 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:51.314133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:51.314066 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:51.314133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:51.314115 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:52.321304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:52.321267 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" event={"ID":"a960f229-1a93-4df8-afd8-025576f0e464","Type":"ContainerStarted","Data":"0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105"} Oct 31 00:44:53.303890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:53.303860 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:44:53.308675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:53.308647 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:44:53.308857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:53.308828 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 31 00:44:53.308993 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:53.308905 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 31 00:44:54.309259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:54.309223 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 31 00:44:54.309752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:54.309296 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 31 00:44:55.308899 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:55.308860 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 31 00:44:55.309201 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:55.308931 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 31 00:44:56.308447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:56.308415 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 31 00:44:56.308920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:56.308470 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 31 00:44:57.308446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:57.308406 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 31 00:44:57.308663 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:57.308464 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 31 00:44:58.312833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:58.312801 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:44:58.313233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:58.313178 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:44:58.313848 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:58.313831 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:44:59.072844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.072805 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm"] Oct 31 00:44:59.073028 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.073002 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" podUID="bcdceb92-a871-45c3-9729-306e0dceb327" containerName="maas-api" containerID="cri-o://f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef" gracePeriod=30 Oct 31 00:44:59.337922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.337864 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:44:59.357144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.357116 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls\") pod \"bcdceb92-a871-45c3-9729-306e0dceb327\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " Oct 31 00:44:59.357303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.357161 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-metrics-certs\") pod \"bcdceb92-a871-45c3-9729-306e0dceb327\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " Oct 31 00:44:59.357303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.357291 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcdceb92-a871-45c3-9729-306e0dceb327-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdceb92-a871-45c3-9729-306e0dceb327-openshift-service-ca\") pod \"bcdceb92-a871-45c3-9729-306e0dceb327\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " Oct 31 00:44:59.357426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.357356 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdceb92-a871-45c3-9729-306e0dceb327-kube-api-access-nldw4\" (UniqueName: \"kubernetes.io/projected/bcdceb92-a871-45c3-9729-306e0dceb327-kube-api-access-nldw4\") pod \"bcdceb92-a871-45c3-9729-306e0dceb327\" (UID: \"bcdceb92-a871-45c3-9729-306e0dceb327\") " Oct 31 00:44:59.357679 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.357649 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcdceb92-a871-45c3-9729-306e0dceb327-openshift-service-ca" pod "bcdceb92-a871-45c3-9729-306e0dceb327" (UID: "bcdceb92-a871-45c3-9729-306e0dceb327"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:59.357977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.357776 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcdceb92-a871-45c3-9729-306e0dceb327-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:59.359338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.359315 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls" pod "bcdceb92-a871-45c3-9729-306e0dceb327" (UID: "bcdceb92-a871-45c3-9729-306e0dceb327"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:59.359715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.359699 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdceb92-a871-45c3-9729-306e0dceb327-kube-api-access-nldw4" pod "bcdceb92-a871-45c3-9729-306e0dceb327" (UID: "bcdceb92-a871-45c3-9729-306e0dceb327"). InnerVolumeSpecName "kube-api-access-nldw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:59.359830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.359816 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-metrics-certs" pod "bcdceb92-a871-45c3-9729-306e0dceb327" (UID: "bcdceb92-a871-45c3-9729-306e0dceb327"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:59.374560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.374535 4969 generic.go:356] "Generic (PLEG): container finished" podID="bcdceb92-a871-45c3-9729-306e0dceb327" containerID="f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef" exitCode=0 Oct 31 00:44:59.374689 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.374579 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" Oct 31 00:44:59.374689 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.374626 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" event={"ID":"bcdceb92-a871-45c3-9729-306e0dceb327","Type":"ContainerDied","Data":"f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef"} Oct 31 00:44:59.374809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.374687 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm" event={"ID":"bcdceb92-a871-45c3-9729-306e0dceb327","Type":"ContainerDied","Data":"e10f166f0af91cf96e687f70a65a14b447d064c33d982cdc0e9fafd9fd377a9c"} Oct 31 00:44:59.374809 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.374718 4969 scope.go:122] "RemoveContainer" containerID="f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef" Oct 31 00:44:59.386477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.386460 4969 scope.go:122] "RemoveContainer" containerID="f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef" Oct 31 00:44:59.386765 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:44:59.386746 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef\": container with ID starting with f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef not found: ID does not exist" containerID="f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef" Oct 31 00:44:59.386827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.386778 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef"} err="failed to get container status \"f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef\": rpc error: code = NotFound desc = could not find container \"f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef\": container with ID starting with f5adf50f11fbdb43b2c7f81c019247292b0190813488c8c715432a9af5f291ef not found: ID does not exist" Oct 31 00:44:59.460079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.459816 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldw4\" (UniqueName: \"kubernetes.io/projected/bcdceb92-a871-45c3-9729-306e0dceb327-kube-api-access-nldw4\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:59.460079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.459851 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:59.460079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.459864 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcdceb92-a871-45c3-9729-306e0dceb327-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:44:59.568961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.568927 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm"] Oct 31 00:44:59.576142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.576117 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-726d5b-58b7d5bf77-wjcsm"] Oct 31 00:44:59.734416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:44:59.734384 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcdceb92-a871-45c3-9729-306e0dceb327" path="/var/lib/kubelet/pods/bcdceb92-a871-45c3-9729-306e0dceb327/volumes" Oct 31 00:45:00.186133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.186091 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng"] Oct 31 00:45:00.186739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.186725 4969 cpu_manager.go:414] "containerMap: removing container" podUID="bcdceb92-a871-45c3-9729-306e0dceb327" containerName="maas-api" Oct 31 00:45:00.186791 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.186741 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcdceb92-a871-45c3-9729-306e0dceb327" containerName="maas-api" Oct 31 00:45:00.186861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.186852 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcdceb92-a871-45c3-9729-306e0dceb327" containerName="maas-api" Oct 31 00:45:00.191764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.191739 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:00.239727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.239698 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t"] Oct 31 00:45:00.244606 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.244571 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:00.246667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.246641 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c"] Oct 31 00:45:00.248697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.248676 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:45:00.251406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.251385 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf"] Oct 31 00:45:00.251531 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.251501 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:00.255772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.255741 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:00.270816 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.270792 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf\" (UID: \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:00.271111 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.271086 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgtts\" (UniqueName: \"kubernetes.io/projected/7b6bae64-5873-424e-ae93-87f387ab5953-kube-api-access-lgtts\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t\" (UID: \"7b6bae64-5873-424e-ae93-87f387ab5953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:00.271272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.271252 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zgx\" (UniqueName: \"kubernetes.io/projected/09a907dd-7f63-47a4-8a5e-3107b264c203-kube-api-access-f9zgx\") pod \"maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c\" (UID: \"09a907dd-7f63-47a4-8a5e-3107b264c203\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:00.271366 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.271349 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkvdt\" (UniqueName: \"kubernetes.io/projected/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-kube-api-access-gkvdt\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf\" (UID: \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:00.271430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.271387 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsxq5\" (UniqueName: \"kubernetes.io/projected/e87bc373-d110-4fda-8e75-dc2c1765b08f-kube-api-access-nsxq5\") pod \"maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng\" (UID: \"e87bc373-d110-4fda-8e75-dc2c1765b08f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:00.271498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.271433 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6bae64-5873-424e-ae93-87f387ab5953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t\" (UID: \"7b6bae64-5873-424e-ae93-87f387ab5953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:00.271498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.271464 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a907dd-7f63-47a4-8a5e-3107b264c203-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c\" (UID: \"09a907dd-7f63-47a4-8a5e-3107b264c203\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:00.271645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.271495 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87bc373-d110-4fda-8e75-dc2c1765b08f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng\" (UID: \"e87bc373-d110-4fda-8e75-dc2c1765b08f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:00.275559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.275534 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng"] Oct 31 00:45:00.321212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.321181 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c"] Oct 31 00:45:00.345722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.345693 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t"] Oct 31 00:45:00.372552 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.372515 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf\" (UID: \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:00.372741 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.372559 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgtts\" (UniqueName: \"kubernetes.io/projected/7b6bae64-5873-424e-ae93-87f387ab5953-kube-api-access-lgtts\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t\" (UID: \"7b6bae64-5873-424e-ae93-87f387ab5953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:00.372741 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.372640 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zgx\" (UniqueName: \"kubernetes.io/projected/09a907dd-7f63-47a4-8a5e-3107b264c203-kube-api-access-f9zgx\") pod \"maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c\" (UID: \"09a907dd-7f63-47a4-8a5e-3107b264c203\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:00.372741 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.372714 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvdt\" (UniqueName: \"kubernetes.io/projected/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-kube-api-access-gkvdt\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf\" (UID: \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:00.372938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.372747 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsxq5\" (UniqueName: \"kubernetes.io/projected/e87bc373-d110-4fda-8e75-dc2c1765b08f-kube-api-access-nsxq5\") pod \"maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng\" (UID: \"e87bc373-d110-4fda-8e75-dc2c1765b08f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:00.372938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.372787 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6bae64-5873-424e-ae93-87f387ab5953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t\" (UID: \"7b6bae64-5873-424e-ae93-87f387ab5953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:00.372938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.372815 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a907dd-7f63-47a4-8a5e-3107b264c203-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c\" (UID: \"09a907dd-7f63-47a4-8a5e-3107b264c203\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:00.372938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.372856 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87bc373-d110-4fda-8e75-dc2c1765b08f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng\" (UID: \"e87bc373-d110-4fda-8e75-dc2c1765b08f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:00.373414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.373385 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf\" (UID: \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:00.373737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.373709 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87bc373-d110-4fda-8e75-dc2c1765b08f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng\" (UID: \"e87bc373-d110-4fda-8e75-dc2c1765b08f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:00.374214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.374192 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a907dd-7f63-47a4-8a5e-3107b264c203-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c\" (UID: \"09a907dd-7f63-47a4-8a5e-3107b264c203\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:00.374313 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.374242 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6bae64-5873-424e-ae93-87f387ab5953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t\" (UID: \"7b6bae64-5873-424e-ae93-87f387ab5953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:00.384293 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.384265 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf"] Oct 31 00:45:00.438139 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.438071 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgtts\" (UniqueName: \"kubernetes.io/projected/7b6bae64-5873-424e-ae93-87f387ab5953-kube-api-access-lgtts\") pod \"maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t\" (UID: \"7b6bae64-5873-424e-ae93-87f387ab5953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:00.444745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.444707 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvdt\" (UniqueName: \"kubernetes.io/projected/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-kube-api-access-gkvdt\") pod \"maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf\" (UID: \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:00.444892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.444803 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zgx\" (UniqueName: \"kubernetes.io/projected/09a907dd-7f63-47a4-8a5e-3107b264c203-kube-api-access-f9zgx\") pod \"maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c\" (UID: \"09a907dd-7f63-47a4-8a5e-3107b264c203\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:00.444942 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.444917 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsxq5\" (UniqueName: \"kubernetes.io/projected/e87bc373-d110-4fda-8e75-dc2c1765b08f-kube-api-access-nsxq5\") pod \"maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng\" (UID: \"e87bc373-d110-4fda-8e75-dc2c1765b08f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:00.505359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.505323 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:00.560031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.559996 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:00.575836 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.575433 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:00.597783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.581814 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:00.931385 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:00.931356 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng"] Oct 31 00:45:00.932381 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:45:00.932351 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87bc373_d110_4fda_8e75_dc2c1765b08f.slice/crio-91022908a2eae210134da5825a3da31b6e8c13f106deed5accb4a88cad4c10e0 WatchSource:0}: Error finding container 91022908a2eae210134da5825a3da31b6e8c13f106deed5accb4a88cad4c10e0: Status 404 returned error can't find the container with id 91022908a2eae210134da5825a3da31b6e8c13f106deed5accb4a88cad4c10e0 Oct 31 00:45:01.035335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:01.035309 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t"] Oct 31 00:45:01.128631 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:45:01.127465 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6bae64_5873_424e_ae93_87f387ab5953.slice/crio-968550296d70ee9f92fb4c632bc56ec806a573ceacc7074a0662df4141c2f433 WatchSource:0}: Error finding container 968550296d70ee9f92fb4c632bc56ec806a573ceacc7074a0662df4141c2f433: Status 404 returned error can't find the container with id 968550296d70ee9f92fb4c632bc56ec806a573ceacc7074a0662df4141c2f433 Oct 31 00:45:01.148716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:01.143868 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf"] Oct 31 00:45:01.154872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:01.154846 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c"] Oct 31 00:45:01.413537 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:01.413509 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" event={"ID":"09a907dd-7f63-47a4-8a5e-3107b264c203","Type":"ContainerStarted","Data":"54f7393259ee26cbe51438f4599831f710800cfd4795a648f367ed8a50ad32ea"} Oct 31 00:45:01.415200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:01.415173 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" event={"ID":"7b6bae64-5873-424e-ae93-87f387ab5953","Type":"ContainerStarted","Data":"968550296d70ee9f92fb4c632bc56ec806a573ceacc7074a0662df4141c2f433"} Oct 31 00:45:01.416456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:01.416427 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" event={"ID":"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1","Type":"ContainerStarted","Data":"c517dae20cfd7c3808db2463be940b8eb012d8dc8bc46a2272105bbc341f70ea"} Oct 31 00:45:01.418319 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:01.418298 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" event={"ID":"e87bc373-d110-4fda-8e75-dc2c1765b08f","Type":"ContainerStarted","Data":"91022908a2eae210134da5825a3da31b6e8c13f106deed5accb4a88cad4c10e0"} Oct 31 00:45:02.428722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.428681 4969 generic.go:356] "Generic (PLEG): container finished" podID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerID="42b24a711c59ee6df2840621b9fb8a3346da9887349c26b452df9dfe8f620823" exitCode=60 Oct 31 00:45:02.429215 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.428750 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" event={"ID":"09a907dd-7f63-47a4-8a5e-3107b264c203","Type":"ContainerDied","Data":"42b24a711c59ee6df2840621b9fb8a3346da9887349c26b452df9dfe8f620823"} Oct 31 00:45:02.429215 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.429113 4969 scope.go:122] "RemoveContainer" containerID="42b24a711c59ee6df2840621b9fb8a3346da9887349c26b452df9dfe8f620823" Oct 31 00:45:02.430585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.430564 4969 generic.go:356] "Generic (PLEG): container finished" podID="7b6bae64-5873-424e-ae93-87f387ab5953" containerID="c6cee091088f15effc3b0399f695e4e13114dcb45b3e9d3ac6ec02bbd133c157" exitCode=60 Oct 31 00:45:02.430750 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.430698 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" event={"ID":"7b6bae64-5873-424e-ae93-87f387ab5953","Type":"ContainerDied","Data":"c6cee091088f15effc3b0399f695e4e13114dcb45b3e9d3ac6ec02bbd133c157"} Oct 31 00:45:02.430932 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.430896 4969 scope.go:122] "RemoveContainer" containerID="c6cee091088f15effc3b0399f695e4e13114dcb45b3e9d3ac6ec02bbd133c157" Oct 31 00:45:02.432799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.432777 4969 generic.go:356] "Generic (PLEG): container finished" podID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerID="05c280d8466a5a87f8a5852f9c699a0fe85d79b23540a5d39d011caf4df51fb5" exitCode=60 Oct 31 00:45:02.433190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.432850 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" event={"ID":"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1","Type":"ContainerDied","Data":"05c280d8466a5a87f8a5852f9c699a0fe85d79b23540a5d39d011caf4df51fb5"} Oct 31 00:45:02.433190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.433077 4969 scope.go:122] "RemoveContainer" containerID="05c280d8466a5a87f8a5852f9c699a0fe85d79b23540a5d39d011caf4df51fb5" Oct 31 00:45:02.434897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.434875 4969 generic.go:356] "Generic (PLEG): container finished" podID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerID="0525e3e0fdff04ff03bc0c3628f474c07ed05599d9cc793b80d01e75e6601f3c" exitCode=60 Oct 31 00:45:02.435001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.434951 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" event={"ID":"e87bc373-d110-4fda-8e75-dc2c1765b08f","Type":"ContainerDied","Data":"0525e3e0fdff04ff03bc0c3628f474c07ed05599d9cc793b80d01e75e6601f3c"} Oct 31 00:45:02.435122 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:02.435110 4969 scope.go:122] "RemoveContainer" containerID="0525e3e0fdff04ff03bc0c3628f474c07ed05599d9cc793b80d01e75e6601f3c" Oct 31 00:45:02.752982 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:02.752223 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:45:03.443847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:03.443809 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" event={"ID":"09a907dd-7f63-47a4-8a5e-3107b264c203","Type":"ContainerStarted","Data":"1883273dd4e87d015c703e2cfce694ef480ca00a97250542fc129d8bbaa58b44"} Oct 31 00:45:03.446026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:03.445992 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" event={"ID":"7b6bae64-5873-424e-ae93-87f387ab5953","Type":"ContainerStarted","Data":"bdf504402dc28ee389984b55deeeddec2fbadbd19de0dd45674e9c2e6cd0d3f0"} Oct 31 00:45:03.447915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:03.447892 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" event={"ID":"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1","Type":"ContainerStarted","Data":"9b6e73eab0636baecff80ffc998f9c5fe394e8b190e023580badfc8622716ec9"} Oct 31 00:45:03.449708 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:03.449682 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" event={"ID":"e87bc373-d110-4fda-8e75-dc2c1765b08f","Type":"ContainerStarted","Data":"13825f4abf391ad0dbc1378698d0e96095246b81a848cf742433e05d15d8543c"} Oct 31 00:45:03.565184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:03.565127 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" podStartSLOduration=3.565113996 podStartE2EDuration="3.565113996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.560691142 +0000 UTC m=+2536.036049567" watchObservedRunningTime="2026-10-31 00:45:03.565113996 +0000 UTC m=+2536.040472419" Oct 31 00:45:03.703072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:03.702942 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" podStartSLOduration=3.702922146 podStartE2EDuration="3.702922146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.697958374 +0000 UTC m=+2536.173316798" watchObservedRunningTime="2026-10-31 00:45:03.702922146 +0000 UTC m=+2536.178280569" Oct 31 00:45:03.817608 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:03.817537 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" podStartSLOduration=3.817522456 podStartE2EDuration="3.817522456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.814185809 +0000 UTC m=+2536.289544229" watchObservedRunningTime="2026-10-31 00:45:03.817522456 +0000 UTC m=+2536.292880878" Oct 31 00:45:03.929843 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:03.929793 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" podStartSLOduration=3.929778308 podStartE2EDuration="3.929778308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.928305238 +0000 UTC m=+2536.403663660" watchObservedRunningTime="2026-10-31 00:45:03.929778308 +0000 UTC m=+2536.405136747" Oct 31 00:45:04.459126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.459089 4969 generic.go:356] "Generic (PLEG): container finished" podID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerID="1883273dd4e87d015c703e2cfce694ef480ca00a97250542fc129d8bbaa58b44" exitCode=60 Oct 31 00:45:04.459648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.459129 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" event={"ID":"09a907dd-7f63-47a4-8a5e-3107b264c203","Type":"ContainerDied","Data":"1883273dd4e87d015c703e2cfce694ef480ca00a97250542fc129d8bbaa58b44"} Oct 31 00:45:04.459648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.459187 4969 scope.go:122] "RemoveContainer" containerID="42b24a711c59ee6df2840621b9fb8a3346da9887349c26b452df9dfe8f620823" Oct 31 00:45:04.459648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.459430 4969 scope.go:122] "RemoveContainer" containerID="1883273dd4e87d015c703e2cfce694ef480ca00a97250542fc129d8bbaa58b44" Oct 31 00:45:04.459799 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:04.459769 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c_odh-ai-gateway-infra(09a907dd-7f63-47a4-8a5e-3107b264c203)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" Oct 31 00:45:04.461306 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.461287 4969 generic.go:356] "Generic (PLEG): container finished" podID="7b6bae64-5873-424e-ae93-87f387ab5953" containerID="bdf504402dc28ee389984b55deeeddec2fbadbd19de0dd45674e9c2e6cd0d3f0" exitCode=60 Oct 31 00:45:04.461439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.461374 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" event={"ID":"7b6bae64-5873-424e-ae93-87f387ab5953","Type":"ContainerDied","Data":"bdf504402dc28ee389984b55deeeddec2fbadbd19de0dd45674e9c2e6cd0d3f0"} Oct 31 00:45:04.461576 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.461561 4969 scope.go:122] "RemoveContainer" containerID="bdf504402dc28ee389984b55deeeddec2fbadbd19de0dd45674e9c2e6cd0d3f0" Oct 31 00:45:04.461906 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:04.461879 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t_odh-ai-gateway-infra(7b6bae64-5873-424e-ae93-87f387ab5953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" Oct 31 00:45:04.463235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.463214 4969 generic.go:356] "Generic (PLEG): container finished" podID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerID="9b6e73eab0636baecff80ffc998f9c5fe394e8b190e023580badfc8622716ec9" exitCode=60 Oct 31 00:45:04.463359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.463282 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" event={"ID":"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1","Type":"ContainerDied","Data":"9b6e73eab0636baecff80ffc998f9c5fe394e8b190e023580badfc8622716ec9"} Oct 31 00:45:04.463508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.463489 4969 scope.go:122] "RemoveContainer" containerID="9b6e73eab0636baecff80ffc998f9c5fe394e8b190e023580badfc8622716ec9" Oct 31 00:45:04.463810 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:04.463784 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf_odh-ai-gateway-infra(4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" Oct 31 00:45:04.465354 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.465332 4969 generic.go:356] "Generic (PLEG): container finished" podID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerID="13825f4abf391ad0dbc1378698d0e96095246b81a848cf742433e05d15d8543c" exitCode=60 Oct 31 00:45:04.465457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.465430 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" event={"ID":"e87bc373-d110-4fda-8e75-dc2c1765b08f","Type":"ContainerDied","Data":"13825f4abf391ad0dbc1378698d0e96095246b81a848cf742433e05d15d8543c"} Oct 31 00:45:04.465651 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.465634 4969 scope.go:122] "RemoveContainer" containerID="13825f4abf391ad0dbc1378698d0e96095246b81a848cf742433e05d15d8543c" Oct 31 00:45:04.465877 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:04.465858 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng_odh-ai-gateway-infra(e87bc373-d110-4fda-8e75-dc2c1765b08f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" Oct 31 00:45:04.473141 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.473120 4969 scope.go:122] "RemoveContainer" containerID="c6cee091088f15effc3b0399f695e4e13114dcb45b3e9d3ac6ec02bbd133c157" Oct 31 00:45:04.511393 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.511373 4969 scope.go:122] "RemoveContainer" containerID="05c280d8466a5a87f8a5852f9c699a0fe85d79b23540a5d39d011caf4df51fb5" Oct 31 00:45:04.524989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:04.524967 4969 scope.go:122] "RemoveContainer" containerID="0525e3e0fdff04ff03bc0c3628f474c07ed05599d9cc793b80d01e75e6601f3c" Oct 31 00:45:05.474959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:05.474930 4969 scope.go:122] "RemoveContainer" containerID="1883273dd4e87d015c703e2cfce694ef480ca00a97250542fc129d8bbaa58b44" Oct 31 00:45:05.475432 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:05.475218 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c_odh-ai-gateway-infra(09a907dd-7f63-47a4-8a5e-3107b264c203)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" Oct 31 00:45:05.496618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:05.476583 4969 scope.go:122] "RemoveContainer" containerID="bdf504402dc28ee389984b55deeeddec2fbadbd19de0dd45674e9c2e6cd0d3f0" Oct 31 00:45:05.496618 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:05.476892 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t_odh-ai-gateway-infra(7b6bae64-5873-424e-ae93-87f387ab5953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" Oct 31 00:45:05.503446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:05.503385 4969 scope.go:122] "RemoveContainer" containerID="9b6e73eab0636baecff80ffc998f9c5fe394e8b190e023580badfc8622716ec9" Oct 31 00:45:05.503761 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:05.503736 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf_odh-ai-gateway-infra(4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" Oct 31 00:45:05.505302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:05.505279 4969 scope.go:122] "RemoveContainer" containerID="13825f4abf391ad0dbc1378698d0e96095246b81a848cf742433e05d15d8543c" Oct 31 00:45:05.505631 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:05.505584 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng_odh-ai-gateway-infra(e87bc373-d110-4fda-8e75-dc2c1765b08f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" Oct 31 00:45:08.920769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:08.920731 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:45:08.921195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:08.920798 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:10.545435 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:10.545404 4969 generic.go:356] "Generic (PLEG): container finished" podID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerID="3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4" exitCode=0 Oct 31 00:45:10.545916 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:10.545478 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" event={"ID":"5e659e33-767b-45ce-98b6-3defaa5924c1","Type":"ContainerDied","Data":"3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4"} Oct 31 00:45:10.545983 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:10.545969 4969 scope.go:122] "RemoveContainer" containerID="3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4" Oct 31 00:45:10.547670 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:10.547642 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:10.547750 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:10.547714 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:10.547790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:10.547752 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:11.555454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:11.555422 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" event={"ID":"5e659e33-767b-45ce-98b6-3defaa5924c1","Type":"ContainerStarted","Data":"226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab"} Oct 31 00:45:12.163506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:12.163473 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:45:12.163811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:12.163789 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:45:12.163911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:12.163844 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 31 00:45:13.163910 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:13.163877 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:45:13.164321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:13.163947 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 31 00:45:14.163475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:14.163441 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:45:14.163690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:14.163491 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 31 00:45:14.727255 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:14.727218 4969 scope.go:122] "RemoveContainer" containerID="13825f4abf391ad0dbc1378698d0e96095246b81a848cf742433e05d15d8543c" Oct 31 00:45:14.727753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:14.727652 4969 scope.go:122] "RemoveContainer" containerID="bdf504402dc28ee389984b55deeeddec2fbadbd19de0dd45674e9c2e6cd0d3f0" Oct 31 00:45:15.163955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:15.163923 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:45:15.164119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:15.163975 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 31 00:45:15.608715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:15.608625 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" event={"ID":"7b6bae64-5873-424e-ae93-87f387ab5953","Type":"ContainerStarted","Data":"1616743b959524cef2b46b8a70d02d7f2f3efb189b768dbc17226873159396c8"} Oct 31 00:45:15.610654 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:15.610625 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" event={"ID":"e87bc373-d110-4fda-8e75-dc2c1765b08f","Type":"ContainerStarted","Data":"0312afff26fb3b22fe5521ff0b086d18d2d61cde119b8a0c78f150b28444df01"} Oct 31 00:45:15.728004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:15.727975 4969 scope.go:122] "RemoveContainer" containerID="1883273dd4e87d015c703e2cfce694ef480ca00a97250542fc129d8bbaa58b44" Oct 31 00:45:16.164367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.164298 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 31 00:45:16.164367 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.164354 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 31 00:45:16.621628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.621576 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" event={"ID":"09a907dd-7f63-47a4-8a5e-3107b264c203","Type":"ContainerStarted","Data":"76a496c6c2f15b00ada9198eef387b61ccddef214bbf4d246c203d383161b2c8"} Oct 31 00:45:16.623908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.623879 4969 generic.go:356] "Generic (PLEG): container finished" podID="7b6bae64-5873-424e-ae93-87f387ab5953" containerID="1616743b959524cef2b46b8a70d02d7f2f3efb189b768dbc17226873159396c8" exitCode=60 Oct 31 00:45:16.624038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.623948 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" event={"ID":"7b6bae64-5873-424e-ae93-87f387ab5953","Type":"ContainerDied","Data":"1616743b959524cef2b46b8a70d02d7f2f3efb189b768dbc17226873159396c8"} Oct 31 00:45:16.624038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.624006 4969 scope.go:122] "RemoveContainer" containerID="bdf504402dc28ee389984b55deeeddec2fbadbd19de0dd45674e9c2e6cd0d3f0" Oct 31 00:45:16.624413 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.624392 4969 scope.go:122] "RemoveContainer" containerID="1616743b959524cef2b46b8a70d02d7f2f3efb189b768dbc17226873159396c8" Oct 31 00:45:16.624805 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:16.624781 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t_odh-ai-gateway-infra(7b6bae64-5873-424e-ae93-87f387ab5953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" Oct 31 00:45:16.626183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.626162 4969 generic.go:356] "Generic (PLEG): container finished" podID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerID="0312afff26fb3b22fe5521ff0b086d18d2d61cde119b8a0c78f150b28444df01" exitCode=60 Oct 31 00:45:16.626293 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.626218 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" event={"ID":"e87bc373-d110-4fda-8e75-dc2c1765b08f","Type":"ContainerDied","Data":"0312afff26fb3b22fe5521ff0b086d18d2d61cde119b8a0c78f150b28444df01"} Oct 31 00:45:16.626544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.626531 4969 scope.go:122] "RemoveContainer" containerID="0312afff26fb3b22fe5521ff0b086d18d2d61cde119b8a0c78f150b28444df01" Oct 31 00:45:16.626813 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:16.626796 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng_odh-ai-gateway-infra(e87bc373-d110-4fda-8e75-dc2c1765b08f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" Oct 31 00:45:16.641281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.639415 4969 scope.go:122] "RemoveContainer" containerID="13825f4abf391ad0dbc1378698d0e96095246b81a848cf742433e05d15d8543c" Oct 31 00:45:16.727930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.727878 4969 scope.go:122] "RemoveContainer" containerID="9b6e73eab0636baecff80ffc998f9c5fe394e8b190e023580badfc8622716ec9" Oct 31 00:45:16.736307 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.736279 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t"] Oct 31 00:45:16.850097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.850070 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng"] Oct 31 00:45:16.950776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.950707 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf"] Oct 31 00:45:16.954817 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:16.954800 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.006696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.006665 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:45:17.011446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.011420 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf"] Oct 31 00:45:17.132924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.132874 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1a05595-hgdrf\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.132924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.132929 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1a05595-hgdrf\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.133220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.133201 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htbsz\" (UniqueName: \"kubernetes.io/projected/c3cb07f1-5589-4157-978b-f25e1ca09640-kube-api-access-htbsz\") pod \"maas-api-revoke-sub-b1a05595-hgdrf\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.168093 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.168068 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:45:17.168547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.168529 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:45:17.169311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.169293 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:45:17.234565 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.234481 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htbsz\" (UniqueName: \"kubernetes.io/projected/c3cb07f1-5589-4157-978b-f25e1ca09640-kube-api-access-htbsz\") pod \"maas-api-revoke-sub-b1a05595-hgdrf\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.234565 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.234561 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1a05595-hgdrf\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.234828 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.234581 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1a05595-hgdrf\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.235343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.235313 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1a05595-hgdrf\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.237503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.237480 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1a05595-hgdrf\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.297235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.297186 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htbsz\" (UniqueName: \"kubernetes.io/projected/c3cb07f1-5589-4157-978b-f25e1ca09640-kube-api-access-htbsz\") pod \"maas-api-revoke-sub-b1a05595-hgdrf\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.569217 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.569144 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:17.640785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.639761 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" event={"ID":"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1","Type":"ContainerStarted","Data":"85107fc66663cb9f54b8b245220dae15f77a5afcb32405323d293ec404970823"} Oct 31 00:45:17.654024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.653987 4969 generic.go:356] "Generic (PLEG): container finished" podID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerID="76a496c6c2f15b00ada9198eef387b61ccddef214bbf4d246c203d383161b2c8" exitCode=60 Oct 31 00:45:17.655331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.654162 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" event={"ID":"09a907dd-7f63-47a4-8a5e-3107b264c203","Type":"ContainerDied","Data":"76a496c6c2f15b00ada9198eef387b61ccddef214bbf4d246c203d383161b2c8"} Oct 31 00:45:17.655331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.654267 4969 scope.go:122] "RemoveContainer" containerID="1883273dd4e87d015c703e2cfce694ef480ca00a97250542fc129d8bbaa58b44" Oct 31 00:45:17.655331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.654806 4969 scope.go:122] "RemoveContainer" containerID="76a496c6c2f15b00ada9198eef387b61ccddef214bbf4d246c203d383161b2c8" Oct 31 00:45:17.655331 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:17.655275 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c_odh-ai-gateway-infra(09a907dd-7f63-47a4-8a5e-3107b264c203)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" Oct 31 00:45:17.756826 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.756457 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c"] Oct 31 00:45:17.889405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.889378 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:17.893214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.893190 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:17.949570 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:17.949537 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf"] Oct 31 00:45:17.950429 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:45:17.950402 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cb07f1_5589_4157_978b_f25e1ca09640.slice/crio-9b6aa49fe478b4c481038b972700aeff5ee85a5ebe2ee87fb97e43cad202ed6a WatchSource:0}: Error finding container 9b6aa49fe478b4c481038b972700aeff5ee85a5ebe2ee87fb97e43cad202ed6a: Status 404 returned error can't find the container with id 9b6aa49fe478b4c481038b972700aeff5ee85a5ebe2ee87fb97e43cad202ed6a Oct 31 00:45:18.051215 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.051180 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87bc373-d110-4fda-8e75-dc2c1765b08f-kube-api-access-nsxq5\" (UniqueName: \"kubernetes.io/projected/e87bc373-d110-4fda-8e75-dc2c1765b08f-kube-api-access-nsxq5\") pod \"e87bc373-d110-4fda-8e75-dc2c1765b08f\" (UID: \"e87bc373-d110-4fda-8e75-dc2c1765b08f\") " Oct 31 00:45:18.051403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.051235 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6bae64-5873-424e-ae93-87f387ab5953-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6bae64-5873-424e-ae93-87f387ab5953-maas-api-service-ca\") pod \"7b6bae64-5873-424e-ae93-87f387ab5953\" (UID: \"7b6bae64-5873-424e-ae93-87f387ab5953\") " Oct 31 00:45:18.051403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.051308 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87bc373-d110-4fda-8e75-dc2c1765b08f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87bc373-d110-4fda-8e75-dc2c1765b08f-maas-api-service-ca\") pod \"e87bc373-d110-4fda-8e75-dc2c1765b08f\" (UID: \"e87bc373-d110-4fda-8e75-dc2c1765b08f\") " Oct 31 00:45:18.051521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.051485 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6bae64-5873-424e-ae93-87f387ab5953-kube-api-access-lgtts\" (UniqueName: \"kubernetes.io/projected/7b6bae64-5873-424e-ae93-87f387ab5953-kube-api-access-lgtts\") pod \"7b6bae64-5873-424e-ae93-87f387ab5953\" (UID: \"7b6bae64-5873-424e-ae93-87f387ab5953\") " Oct 31 00:45:18.051681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.051659 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6bae64-5873-424e-ae93-87f387ab5953-maas-api-service-ca" pod "7b6bae64-5873-424e-ae93-87f387ab5953" (UID: "7b6bae64-5873-424e-ae93-87f387ab5953"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.051753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.051707 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87bc373-d110-4fda-8e75-dc2c1765b08f-maas-api-service-ca" pod "e87bc373-d110-4fda-8e75-dc2c1765b08f" (UID: "e87bc373-d110-4fda-8e75-dc2c1765b08f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.051969 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.051951 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6bae64-5873-424e-ae93-87f387ab5953-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.052011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.051973 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87bc373-d110-4fda-8e75-dc2c1765b08f-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.053720 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.053702 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6bae64-5873-424e-ae93-87f387ab5953-kube-api-access-lgtts" pod "7b6bae64-5873-424e-ae93-87f387ab5953" (UID: "7b6bae64-5873-424e-ae93-87f387ab5953"). InnerVolumeSpecName "kube-api-access-lgtts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.053821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.053801 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87bc373-d110-4fda-8e75-dc2c1765b08f-kube-api-access-nsxq5" pod "e87bc373-d110-4fda-8e75-dc2c1765b08f" (UID: "e87bc373-d110-4fda-8e75-dc2c1765b08f"). InnerVolumeSpecName "kube-api-access-nsxq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.152734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.152700 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgtts\" (UniqueName: \"kubernetes.io/projected/7b6bae64-5873-424e-ae93-87f387ab5953-kube-api-access-lgtts\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.152734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.152726 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsxq5\" (UniqueName: \"kubernetes.io/projected/e87bc373-d110-4fda-8e75-dc2c1765b08f-kube-api-access-nsxq5\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.670521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.670497 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" Oct 31 00:45:18.670521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.670516 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t" event={"ID":"7b6bae64-5873-424e-ae93-87f387ab5953","Type":"ContainerDied","Data":"968550296d70ee9f92fb4c632bc56ec806a573ceacc7074a0662df4141c2f433"} Oct 31 00:45:18.670747 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.670574 4969 scope.go:122] "RemoveContainer" containerID="1616743b959524cef2b46b8a70d02d7f2f3efb189b768dbc17226873159396c8" Oct 31 00:45:18.672710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.672671 4969 generic.go:356] "Generic (PLEG): container finished" podID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerID="85107fc66663cb9f54b8b245220dae15f77a5afcb32405323d293ec404970823" exitCode=60 Oct 31 00:45:18.672829 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.672742 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" event={"ID":"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1","Type":"ContainerDied","Data":"85107fc66663cb9f54b8b245220dae15f77a5afcb32405323d293ec404970823"} Oct 31 00:45:18.673148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.673129 4969 scope.go:122] "RemoveContainer" containerID="85107fc66663cb9f54b8b245220dae15f77a5afcb32405323d293ec404970823" Oct 31 00:45:18.673440 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:18.673416 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf_odh-ai-gateway-infra(4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" Oct 31 00:45:18.674446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.674419 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" Oct 31 00:45:18.674609 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.674471 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng" event={"ID":"e87bc373-d110-4fda-8e75-dc2c1765b08f","Type":"ContainerDied","Data":"91022908a2eae210134da5825a3da31b6e8c13f106deed5accb4a88cad4c10e0"} Oct 31 00:45:18.681360 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.681329 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" event={"ID":"c3cb07f1-5589-4157-978b-f25e1ca09640","Type":"ContainerStarted","Data":"97dc8ec6ab54250a8795d16b7cfae23a00eed2193fe6abce399c0906da02c579"} Oct 31 00:45:18.681469 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.681372 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" event={"ID":"c3cb07f1-5589-4157-978b-f25e1ca09640","Type":"ContainerStarted","Data":"9b6aa49fe478b4c481038b972700aeff5ee85a5ebe2ee87fb97e43cad202ed6a"} Oct 31 00:45:18.683421 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.683405 4969 scope.go:122] "RemoveContainer" containerID="9b6e73eab0636baecff80ffc998f9c5fe394e8b190e023580badfc8622716ec9" Oct 31 00:45:18.700076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.700056 4969 scope.go:122] "RemoveContainer" containerID="0312afff26fb3b22fe5521ff0b086d18d2d61cde119b8a0c78f150b28444df01" Oct 31 00:45:18.780238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.780203 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf"] Oct 31 00:45:18.840973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.840947 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:18.902491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.902438 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" podStartSLOduration=2.902424098 podStartE2EDuration="2.902424098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:18.900954132 +0000 UTC m=+2551.376312557" watchObservedRunningTime="2026-10-31 00:45:18.902424098 +0000 UTC m=+2551.377782520" Oct 31 00:45:18.963465 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.963385 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a907dd-7f63-47a4-8a5e-3107b264c203-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a907dd-7f63-47a4-8a5e-3107b264c203-maas-api-service-ca\") pod \"09a907dd-7f63-47a4-8a5e-3107b264c203\" (UID: \"09a907dd-7f63-47a4-8a5e-3107b264c203\") " Oct 31 00:45:18.963648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.963581 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a907dd-7f63-47a4-8a5e-3107b264c203-kube-api-access-f9zgx\" (UniqueName: \"kubernetes.io/projected/09a907dd-7f63-47a4-8a5e-3107b264c203-kube-api-access-f9zgx\") pod \"09a907dd-7f63-47a4-8a5e-3107b264c203\" (UID: \"09a907dd-7f63-47a4-8a5e-3107b264c203\") " Oct 31 00:45:18.963734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.963713 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a907dd-7f63-47a4-8a5e-3107b264c203-maas-api-service-ca" pod "09a907dd-7f63-47a4-8a5e-3107b264c203" (UID: "09a907dd-7f63-47a4-8a5e-3107b264c203"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.963987 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.963969 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a907dd-7f63-47a4-8a5e-3107b264c203-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.965865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:18.965844 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a907dd-7f63-47a4-8a5e-3107b264c203-kube-api-access-f9zgx" pod "09a907dd-7f63-47a4-8a5e-3107b264c203" (UID: "09a907dd-7f63-47a4-8a5e-3107b264c203"). InnerVolumeSpecName "kube-api-access-f9zgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:19.064882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.064854 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9zgx\" (UniqueName: \"kubernetes.io/projected/09a907dd-7f63-47a4-8a5e-3107b264c203-kube-api-access-f9zgx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.129755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.129723 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng"] Oct 31 00:45:19.134235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.134211 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-87b0f8-29890125-j62ng"] Oct 31 00:45:19.245541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.245464 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t"] Oct 31 00:45:19.252799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.252773 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7abb95-29890125-92d5t"] Oct 31 00:45:19.693379 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.693346 4969 generic.go:356] "Generic (PLEG): container finished" podID="c3cb07f1-5589-4157-978b-f25e1ca09640" containerID="97dc8ec6ab54250a8795d16b7cfae23a00eed2193fe6abce399c0906da02c579" exitCode=0 Oct 31 00:45:19.693538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.693434 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" event={"ID":"c3cb07f1-5589-4157-978b-f25e1ca09640","Type":"ContainerDied","Data":"97dc8ec6ab54250a8795d16b7cfae23a00eed2193fe6abce399c0906da02c579"} Oct 31 00:45:19.694984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.694964 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" Oct 31 00:45:19.695115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.694993 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c" event={"ID":"09a907dd-7f63-47a4-8a5e-3107b264c203","Type":"ContainerDied","Data":"54f7393259ee26cbe51438f4599831f710800cfd4795a648f367ed8a50ad32ea"} Oct 31 00:45:19.695115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.695051 4969 scope.go:122] "RemoveContainer" containerID="76a496c6c2f15b00ada9198eef387b61ccddef214bbf4d246c203d383161b2c8" Oct 31 00:45:19.736821 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.736794 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" path="/var/lib/kubelet/pods/7b6bae64-5873-424e-ae93-87f387ab5953/volumes" Oct 31 00:45:19.737325 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.737304 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" path="/var/lib/kubelet/pods/e87bc373-d110-4fda-8e75-dc2c1765b08f/volumes" Oct 31 00:45:19.861377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.861350 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:19.926039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.925999 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c"] Oct 31 00:45:19.932195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.932165 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-868b06-29890125-htc4c"] Oct 31 00:45:19.976340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.976264 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-maas-api-service-ca\") pod \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\" (UID: \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\") " Oct 31 00:45:19.976340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.976326 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-kube-api-access-gkvdt\" (UniqueName: \"kubernetes.io/projected/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-kube-api-access-gkvdt\") pod \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\" (UID: \"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1\") " Oct 31 00:45:19.976697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.976674 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-maas-api-service-ca" pod "4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" (UID: "4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:19.978716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:19.978691 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-kube-api-access-gkvdt" pod "4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" (UID: "4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1"). InnerVolumeSpecName "kube-api-access-gkvdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:20.077871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.077823 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.077871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.077856 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkvdt\" (UniqueName: \"kubernetes.io/projected/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1-kube-api-access-gkvdt\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.705225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.705194 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" Oct 31 00:45:20.705422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.705221 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf" event={"ID":"4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1","Type":"ContainerDied","Data":"c517dae20cfd7c3808db2463be940b8eb012d8dc8bc46a2272105bbc341f70ea"} Oct 31 00:45:20.705422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.705305 4969 scope.go:122] "RemoveContainer" containerID="85107fc66663cb9f54b8b245220dae15f77a5afcb32405323d293ec404970823" Oct 31 00:45:20.872374 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.872343 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf"] Oct 31 00:45:20.872850 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.872430 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:20.877939 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.877910 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e1ca71-29890125-mqgcf"] Oct 31 00:45:20.989764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.989670 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-openshift-ca\") pod \"c3cb07f1-5589-4157-978b-f25e1ca09640\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " Oct 31 00:45:20.990210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.989795 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-xks-ca\") pod \"c3cb07f1-5589-4157-978b-f25e1ca09640\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " Oct 31 00:45:20.990210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.989843 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cb07f1-5589-4157-978b-f25e1ca09640-kube-api-access-htbsz\" (UniqueName: \"kubernetes.io/projected/c3cb07f1-5589-4157-978b-f25e1ca09640-kube-api-access-htbsz\") pod \"c3cb07f1-5589-4157-978b-f25e1ca09640\" (UID: \"c3cb07f1-5589-4157-978b-f25e1ca09640\") " Oct 31 00:45:20.990522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.990478 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-openshift-ca" pod "c3cb07f1-5589-4157-978b-f25e1ca09640" (UID: "c3cb07f1-5589-4157-978b-f25e1ca09640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:20.993142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.993080 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cb07f1-5589-4157-978b-f25e1ca09640-kube-api-access-htbsz" pod "c3cb07f1-5589-4157-978b-f25e1ca09640" (UID: "c3cb07f1-5589-4157-978b-f25e1ca09640"). InnerVolumeSpecName "kube-api-access-htbsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:20.999883 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:20.999857 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-xks-ca" pod "c3cb07f1-5589-4157-978b-f25e1ca09640" (UID: "c3cb07f1-5589-4157-978b-f25e1ca09640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:21.090972 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:21.090946 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:21.091115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:21.090975 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cb07f1-5589-4157-978b-f25e1ca09640-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:21.091115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:21.090991 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htbsz\" (UniqueName: \"kubernetes.io/projected/c3cb07f1-5589-4157-978b-f25e1ca09640-kube-api-access-htbsz\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:45:21.717798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:21.717766 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" Oct 31 00:45:21.717798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:21.717784 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf" event={"ID":"c3cb07f1-5589-4157-978b-f25e1ca09640","Type":"ContainerDied","Data":"9b6aa49fe478b4c481038b972700aeff5ee85a5ebe2ee87fb97e43cad202ed6a"} Oct 31 00:45:21.718014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:21.717833 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b6aa49fe478b4c481038b972700aeff5ee85a5ebe2ee87fb97e43cad202ed6a" Oct 31 00:45:21.734804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:21.734768 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" path="/var/lib/kubelet/pods/09a907dd-7f63-47a4-8a5e-3107b264c203/volumes" Oct 31 00:45:21.735292 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:21.735269 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" path="/var/lib/kubelet/pods/4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1/volumes" Oct 31 00:45:32.729009 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:32.728976 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:45:50.239162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239129 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4"] Oct 31 00:45:50.239783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239753 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerName="cleanup" Oct 31 00:45:50.239783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239773 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239792 4969 cpu_manager.go:414] "containerMap: removing container" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239801 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239813 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239820 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239830 4969 cpu_manager.go:414] "containerMap: removing container" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239834 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239844 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239848 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239864 4969 cpu_manager.go:414] "containerMap: removing container" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239869 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239877 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c3cb07f1-5589-4157-978b-f25e1ca09640" containerName="revoke-keys" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239883 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cb07f1-5589-4157-978b-f25e1ca09640" containerName="revoke-keys" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239890 4969 cpu_manager.go:414] "containerMap: removing container" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239895 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239900 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerName="cleanup" Oct 31 00:45:50.240005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.239905 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240022 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240029 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240034 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240040 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cb07f1-5589-4157-978b-f25e1ca09640" containerName="revoke-keys" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240047 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240051 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240056 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240064 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240071 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240077 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" containerName="cleanup" Oct 31 00:45:50.241069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.240082 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerName="cleanup" Oct 31 00:45:50.245303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.245283 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.308168 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.308144 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ef8827-serving-cert\"" Oct 31 00:45:50.361733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.361703 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4"] Oct 31 00:45:50.420268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.420235 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.420448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.420309 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-metrics-certs\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.420448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.420394 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.420580 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.420565 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6kt\" (UniqueName: \"kubernetes.io/projected/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-kube-api-access-qs6kt\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.521624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.521507 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6kt\" (UniqueName: \"kubernetes.io/projected/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-kube-api-access-qs6kt\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.521624 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.521550 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.521832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.521651 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-metrics-certs\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.521832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.521689 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.521832 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:50.521693 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-serving-cert: secret "maas-api-e2e-praxis-b-ef8827-serving-cert" not found Oct 31 00:45:50.521832 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:45:50.521777 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls podName:cb2a48c7-089d-4ea2-b952-8c3b48d7ab86 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:51.02175322 +0000 UTC m=+2583.497111634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls") pod "maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" (UID: "cb2a48c7-089d-4ea2-b952-8c3b48d7ab86") : secret "maas-api-e2e-praxis-b-ef8827-serving-cert" not found Oct 31 00:45:50.522380 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.522363 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.524648 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.524619 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-metrics-certs\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:50.583200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:50.583164 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6kt\" (UniqueName: \"kubernetes.io/projected/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-kube-api-access-qs6kt\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:51.027074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:51.027036 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:51.030620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:51.030566 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:51.159274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:51.159244 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:51.539915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:51.539884 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4"] Oct 31 00:45:51.541507 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:45:51.541476 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2a48c7_089d_4ea2_b952_8c3b48d7ab86.slice/crio-506954e7f5d59ed3bcd528d190b24ffee611f3390af2ff22076487a0148d1858 WatchSource:0}: Error finding container 506954e7f5d59ed3bcd528d190b24ffee611f3390af2ff22076487a0148d1858: Status 404 returned error can't find the container with id 506954e7f5d59ed3bcd528d190b24ffee611f3390af2ff22076487a0148d1858 Oct 31 00:45:51.960898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:51.959079 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" event={"ID":"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86","Type":"ContainerStarted","Data":"506954e7f5d59ed3bcd528d190b24ffee611f3390af2ff22076487a0148d1858"} Oct 31 00:45:52.968877 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:52.968825 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" event={"ID":"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86","Type":"ContainerStarted","Data":"b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8"} Oct 31 00:45:52.969311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:52.968929 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:45:53.086396 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:53.086331 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" podStartSLOduration=2.210988355 podStartE2EDuration="3.086315096s" totalImagesPullingTime="875.326741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:53.085902134 +0000 UTC m=+2585.561260584" watchObservedRunningTime="2026-10-31 00:45:53.086315096 +0000 UTC m=+2585.561673519" Oct 31 00:45:58.982424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:45:58.982386 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:46:02.728504 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:46:02.728465 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:46:21.499399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:21.499368 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc"] Oct 31 00:46:26.500270 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:26.500216 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" podUID="f33ee314-f517-410c-8dff-bc0ebe2ae914" containerName="payload-processing" containerID="cri-o://5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa" gracePeriod=25 Oct 31 00:46:28.306933 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:28.306892 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" podUID="f33ee314-f517-410c-8dff-bc0ebe2ae914" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:46:30.287001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.286969 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl"] Oct 31 00:46:30.287575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287558 4969 cpu_manager.go:414] "containerMap: removing container" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" containerName="cleanup" Oct 31 00:46:30.287575 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287575 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" containerName="cleanup" Oct 31 00:46:30.287756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287603 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerName="cleanup" Oct 31 00:46:30.287756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287608 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87bc373-d110-4fda-8e75-dc2c1765b08f" containerName="cleanup" Oct 31 00:46:30.287756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287634 4969 cpu_manager.go:414] "containerMap: removing container" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerName="cleanup" Oct 31 00:46:30.287756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287639 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a907dd-7f63-47a4-8a5e-3107b264c203" containerName="cleanup" Oct 31 00:46:30.287756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287651 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerName="cleanup" Oct 31 00:46:30.287756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287657 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerName="cleanup" Oct 31 00:46:30.288079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287762 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6bae64-5873-424e-ae93-87f387ab5953" containerName="cleanup" Oct 31 00:46:30.288079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.287775 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4225a40f-4fc6-49c0-81d3-aa32bd5ae4b1" containerName="cleanup" Oct 31 00:46:30.291411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.291391 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.343646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.343622 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-31144e-serving-cert\"" Oct 31 00:46:30.397302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.397277 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl"] Oct 31 00:46:30.400131 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.400107 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggc2h\" (UniqueName: \"kubernetes.io/projected/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-kube-api-access-ggc2h\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.400261 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.400149 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.400331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.400315 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.400392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.400351 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-metrics-certs\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.501243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.501210 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-metrics-certs\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.501405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.501294 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggc2h\" (UniqueName: \"kubernetes.io/projected/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-kube-api-access-ggc2h\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.501405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.501344 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.501503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.501473 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.501676 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:46:30.501657 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-serving-cert: secret "maas-api-e2e-praxis-b-31144e-serving-cert" not found Oct 31 00:46:30.501763 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:46:30.501752 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls podName:6a5d428a-0d78-49c6-89f3-62fa31adb3cc nodeName:}" failed. No retries permitted until 2026-10-31 00:46:31.00173075 +0000 UTC m=+2623.477089167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls") pod "maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" (UID: "6a5d428a-0d78-49c6-89f3-62fa31adb3cc") : secret "maas-api-e2e-praxis-b-31144e-serving-cert" not found Oct 31 00:46:30.502089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.502065 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.503903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.503883 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-metrics-certs\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:30.562528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:30.562459 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggc2h\" (UniqueName: \"kubernetes.io/projected/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-kube-api-access-ggc2h\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:31.007270 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:31.007234 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:31.009914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:31.009892 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:31.206158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:31.206126 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:31.590856 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:31.590828 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl"] Oct 31 00:46:31.591656 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:46:31.591629 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5d428a_0d78_49c6_89f3_62fa31adb3cc.slice/crio-d496695cb6f85735809ee1c35b91c034732ea1328700d8d46c77ab989c7c7ba8 WatchSource:0}: Error finding container d496695cb6f85735809ee1c35b91c034732ea1328700d8d46c77ab989c7c7ba8: Status 404 returned error can't find the container with id d496695cb6f85735809ee1c35b91c034732ea1328700d8d46c77ab989c7c7ba8 Oct 31 00:46:32.266133 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:32.266100 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" event={"ID":"6a5d428a-0d78-49c6-89f3-62fa31adb3cc","Type":"ContainerStarted","Data":"d496695cb6f85735809ee1c35b91c034732ea1328700d8d46c77ab989c7c7ba8"} Oct 31 00:46:32.727714 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:46:32.727680 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:46:33.274869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:33.274833 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" event={"ID":"6a5d428a-0d78-49c6-89f3-62fa31adb3cc","Type":"ContainerStarted","Data":"fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90"} Oct 31 00:46:33.275082 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:33.274917 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:33.391814 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:33.391755 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" podStartSLOduration=2.59976302 podStartE2EDuration="3.391740462s" totalImagesPullingTime="791.977442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:33.391508409 +0000 UTC m=+2625.866866833" watchObservedRunningTime="2026-10-31 00:46:33.391740462 +0000 UTC m=+2625.867098885" Oct 31 00:46:38.306635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:38.306582 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" podUID="f33ee314-f517-410c-8dff-bc0ebe2ae914" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:46:39.287771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:39.287742 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:46:46.766823 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:46.766763 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:46:46.893861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:46.893832 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33ee314-f517-410c-8dff-bc0ebe2ae914-kube-api-access-bfsbz\" (UniqueName: \"kubernetes.io/projected/f33ee314-f517-410c-8dff-bc0ebe2ae914-kube-api-access-bfsbz\") pod \"f33ee314-f517-410c-8dff-bc0ebe2ae914\" (UID: \"f33ee314-f517-410c-8dff-bc0ebe2ae914\") " Oct 31 00:46:46.894051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:46.893943 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33ee314-f517-410c-8dff-bc0ebe2ae914-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ee314-f517-410c-8dff-bc0ebe2ae914-plugins-config-volume\") pod \"f33ee314-f517-410c-8dff-bc0ebe2ae914\" (UID: \"f33ee314-f517-410c-8dff-bc0ebe2ae914\") " Oct 31 00:46:46.894339 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:46.894316 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33ee314-f517-410c-8dff-bc0ebe2ae914-plugins-config-volume" pod "f33ee314-f517-410c-8dff-bc0ebe2ae914" (UID: "f33ee314-f517-410c-8dff-bc0ebe2ae914"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:46.896415 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:46.896387 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33ee314-f517-410c-8dff-bc0ebe2ae914-kube-api-access-bfsbz" pod "f33ee314-f517-410c-8dff-bc0ebe2ae914" (UID: "f33ee314-f517-410c-8dff-bc0ebe2ae914"). InnerVolumeSpecName "kube-api-access-bfsbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:46.995454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:46.995424 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ee314-f517-410c-8dff-bc0ebe2ae914-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:46.995454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:46.995452 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfsbz\" (UniqueName: \"kubernetes.io/projected/f33ee314-f517-410c-8dff-bc0ebe2ae914-kube-api-access-bfsbz\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:46:47.377983 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.377949 4969 generic.go:356] "Generic (PLEG): container finished" podID="f33ee314-f517-410c-8dff-bc0ebe2ae914" containerID="5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa" exitCode=0 Oct 31 00:46:47.378163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.377998 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" Oct 31 00:46:47.378163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.378034 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" event={"ID":"f33ee314-f517-410c-8dff-bc0ebe2ae914","Type":"ContainerDied","Data":"5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa"} Oct 31 00:46:47.378163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.378088 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc" event={"ID":"f33ee314-f517-410c-8dff-bc0ebe2ae914","Type":"ContainerDied","Data":"b1a7abc489ee5cecda80ade64c23db0ef7bd1191e05fa50d1db7b2fc5da49c1d"} Oct 31 00:46:47.378163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.378110 4969 scope.go:122] "RemoveContainer" containerID="5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa" Oct 31 00:46:47.389261 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.389244 4969 scope.go:122] "RemoveContainer" containerID="5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa" Oct 31 00:46:47.389546 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:46:47.389526 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa\": container with ID starting with 5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa not found: ID does not exist" containerID="5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa" Oct 31 00:46:47.389644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.389554 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa"} err="failed to get container status \"5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa\": rpc error: code = NotFound desc = could not find container \"5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa\": container with ID starting with 5ed6b14f96a62c547cd9cd4796b7cbc32e4827e93ea08acf34fce28bbcf8e3fa not found: ID does not exist" Oct 31 00:46:47.543916 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.543876 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc"] Oct 31 00:46:47.593891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.593858 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-87b0f8-d456cbfbf-8wrkc"] Oct 31 00:46:47.734723 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:46:47.734653 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33ee314-f517-410c-8dff-bc0ebe2ae914" path="/var/lib/kubelet/pods/f33ee314-f517-410c-8dff-bc0ebe2ae914/volumes" Oct 31 00:47:02.728366 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:47:02.728282 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:47:10.618477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.618445 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6"] Oct 31 00:47:10.619085 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.619069 4969 cpu_manager.go:414] "containerMap: removing container" podUID="f33ee314-f517-410c-8dff-bc0ebe2ae914" containerName="payload-processing" Oct 31 00:47:10.619131 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.619086 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33ee314-f517-410c-8dff-bc0ebe2ae914" containerName="payload-processing" Oct 31 00:47:10.619225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.619216 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33ee314-f517-410c-8dff-bc0ebe2ae914" containerName="payload-processing" Oct 31 00:47:10.624030 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.624011 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.676355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.676330 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e1ca71-openshift-default-dockercfg-sh7lb\"" Oct 31 00:47:10.730871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.730837 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6"] Oct 31 00:47:10.785647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.785574 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6fcx\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-kube-api-access-b6fcx\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.785827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.785721 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.785827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.785749 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.785935 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.785870 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.785935 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.785906 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.786033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.785941 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.786033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.786002 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.786123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.786040 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.786123 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.786106 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887261 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887298 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887359 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887383 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887471 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887521 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887552 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887657 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887760 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fcx\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-kube-api-access-b6fcx\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887821 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.887914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887855 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.888138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.887935 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.888454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.888430 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.888857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.888826 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.890026 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.890007 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.890248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.890228 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.948764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.948732 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:10.948930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:10.948855 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fcx\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-kube-api-access-b6fcx\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:11.237819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:11.237732 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:11.672062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:11.672036 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6"] Oct 31 00:47:11.672762 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:47:11.672738 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81ec4b1_24ea_4bb9_bf45_64badd7bfc54.slice/crio-677c83b4b7d7fe1aa2bfec988f399c6d4a2a78532d2d55be42784c7251810c98 WatchSource:0}: Error finding container 677c83b4b7d7fe1aa2bfec988f399c6d4a2a78532d2d55be42784c7251810c98: Status 404 returned error can't find the container with id 677c83b4b7d7fe1aa2bfec988f399c6d4a2a78532d2d55be42784c7251810c98 Oct 31 00:47:11.675246 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:11.675207 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:11.675349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:11.675277 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:11.675349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:11.675313 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:12.598890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:12.598852 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" event={"ID":"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54","Type":"ContainerStarted","Data":"6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4"} Oct 31 00:47:12.599187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:12.599166 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" event={"ID":"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54","Type":"ContainerStarted","Data":"677c83b4b7d7fe1aa2bfec988f399c6d4a2a78532d2d55be42784c7251810c98"} Oct 31 00:47:12.772804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:12.772746 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" podStartSLOduration=2.772730893 podStartE2EDuration="2.772730893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:12.771801133 +0000 UTC m=+2665.247159556" watchObservedRunningTime="2026-10-31 00:47:12.772730893 +0000 UTC m=+2665.248089315" Oct 31 00:47:13.238974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:13.238941 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:13.240489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:13.240464 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 31 00:47:13.240618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:13.240525 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 31 00:47:14.238329 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:14.238292 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 31 00:47:14.238775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:14.238354 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 31 00:47:15.238815 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:15.238773 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 31 00:47:15.239329 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:15.238853 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 31 00:47:15.243251 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:15.243225 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f"] Oct 31 00:47:15.352150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:15.352116 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv"] Oct 31 00:47:16.238845 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:16.238804 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 31 00:47:16.239488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:16.238894 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 31 00:47:17.238346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:17.238310 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 31 00:47:17.238544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:17.238372 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 31 00:47:18.238989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:18.238952 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 31 00:47:18.239479 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:18.239015 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 31 00:47:19.243498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:19.243464 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:19.244003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:19.243978 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:19.244752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:19.244731 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:47:20.244185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.244141 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" podUID="468b1c7e-5913-4a04-807a-8ea0a8b28506" containerName="payload-processing" containerID="cri-o://ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00" gracePeriod=25 Oct 31 00:47:20.352952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.352910 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" podUID="cd621d4c-6bc9-44bf-9af8-089078325917" containerName="payload-pre-processing" containerID="cri-o://4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d" gracePeriod=25 Oct 31 00:47:20.402616 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:47:20.398264 4969 logging.go:55] [core] [Channel #5558 SubChannel #5559]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.187:50052", ServerName: "10.133.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.187:50052: connect: connection refused" Oct 31 00:47:20.402616 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:47:20.398772 4969 logging.go:55] [core] [Channel #5560 SubChannel #5561]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.188:50052", ServerName: "10.133.0.188:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.188:50052: connect: connection refused" Oct 31 00:47:20.568198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.567815 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:47:20.608607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.608559 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468b1c7e-5913-4a04-807a-8ea0a8b28506-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b1c7e-5913-4a04-807a-8ea0a8b28506-plugins-config-volume\") pod \"468b1c7e-5913-4a04-807a-8ea0a8b28506\" (UID: \"468b1c7e-5913-4a04-807a-8ea0a8b28506\") " Oct 31 00:47:20.608836 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.608815 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468b1c7e-5913-4a04-807a-8ea0a8b28506-kube-api-access-zqlqn\" (UniqueName: \"kubernetes.io/projected/468b1c7e-5913-4a04-807a-8ea0a8b28506-kube-api-access-zqlqn\") pod \"468b1c7e-5913-4a04-807a-8ea0a8b28506\" (UID: \"468b1c7e-5913-4a04-807a-8ea0a8b28506\") " Oct 31 00:47:20.609019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.608994 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468b1c7e-5913-4a04-807a-8ea0a8b28506-plugins-config-volume" pod "468b1c7e-5913-4a04-807a-8ea0a8b28506" (UID: "468b1c7e-5913-4a04-807a-8ea0a8b28506"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:20.609387 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.609366 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b1c7e-5913-4a04-807a-8ea0a8b28506-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:20.612228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.612199 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468b1c7e-5913-4a04-807a-8ea0a8b28506-kube-api-access-zqlqn" pod "468b1c7e-5913-4a04-807a-8ea0a8b28506" (UID: "468b1c7e-5913-4a04-807a-8ea0a8b28506"). InnerVolumeSpecName "kube-api-access-zqlqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:20.640570 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.640547 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:47:20.661203 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.661175 4969 generic.go:356] "Generic (PLEG): container finished" podID="468b1c7e-5913-4a04-807a-8ea0a8b28506" containerID="ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00" exitCode=0 Oct 31 00:47:20.661334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.661223 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" Oct 31 00:47:20.661334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.661265 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" event={"ID":"468b1c7e-5913-4a04-807a-8ea0a8b28506","Type":"ContainerDied","Data":"ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00"} Oct 31 00:47:20.661467 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.661360 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" event={"ID":"468b1c7e-5913-4a04-807a-8ea0a8b28506","Type":"ContainerDied","Data":"f1c595ec50bd3949461eec407a804c825a11da05b5ad137943fbaf005822b92c"} Oct 31 00:47:20.661467 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.661395 4969 scope.go:122] "RemoveContainer" containerID="ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00" Oct 31 00:47:20.662961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.662937 4969 generic.go:356] "Generic (PLEG): container finished" podID="cd621d4c-6bc9-44bf-9af8-089078325917" containerID="4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d" exitCode=0 Oct 31 00:47:20.663072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.662984 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" Oct 31 00:47:20.663072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.663044 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" event={"ID":"cd621d4c-6bc9-44bf-9af8-089078325917","Type":"ContainerDied","Data":"4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d"} Oct 31 00:47:20.663184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.663076 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" event={"ID":"cd621d4c-6bc9-44bf-9af8-089078325917","Type":"ContainerDied","Data":"159a8cefe09671fda3a285bdcbea449fe9a0464a623e7fcc91b478c2be6e5f1c"} Oct 31 00:47:20.676254 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.676230 4969 scope.go:122] "RemoveContainer" containerID="ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00" Oct 31 00:47:20.676547 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:47:20.676528 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00\": container with ID starting with ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00 not found: ID does not exist" containerID="ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00" Oct 31 00:47:20.676627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.676556 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00"} err="failed to get container status \"ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00\": rpc error: code = NotFound desc = could not find container \"ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00\": container with ID starting with ee60841e9e29cfe337d31ec33108c7e77d3188147552f0298eb49ae71b06cb00 not found: ID does not exist" Oct 31 00:47:20.676627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.676574 4969 scope.go:122] "RemoveContainer" containerID="4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d" Oct 31 00:47:20.686585 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.686562 4969 scope.go:122] "RemoveContainer" containerID="4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d" Oct 31 00:47:20.686901 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:47:20.686880 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d\": container with ID starting with 4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d not found: ID does not exist" containerID="4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d" Oct 31 00:47:20.687015 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.686911 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d"} err="failed to get container status \"4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d\": rpc error: code = NotFound desc = could not find container \"4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d\": container with ID starting with 4b810e885793918ef15f061dc36b2aee23e941f2a6388a1c4f721d2a3a637b8d not found: ID does not exist" Oct 31 00:47:20.709938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.709911 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd621d4c-6bc9-44bf-9af8-089078325917-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd621d4c-6bc9-44bf-9af8-089078325917-plugins-config-volume\") pod \"cd621d4c-6bc9-44bf-9af8-089078325917\" (UID: \"cd621d4c-6bc9-44bf-9af8-089078325917\") " Oct 31 00:47:20.710148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.710127 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd621d4c-6bc9-44bf-9af8-089078325917-kube-api-access-gb4qb\" (UniqueName: \"kubernetes.io/projected/cd621d4c-6bc9-44bf-9af8-089078325917-kube-api-access-gb4qb\") pod \"cd621d4c-6bc9-44bf-9af8-089078325917\" (UID: \"cd621d4c-6bc9-44bf-9af8-089078325917\") " Oct 31 00:47:20.710328 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.710307 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd621d4c-6bc9-44bf-9af8-089078325917-plugins-config-volume" pod "cd621d4c-6bc9-44bf-9af8-089078325917" (UID: "cd621d4c-6bc9-44bf-9af8-089078325917"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:20.710846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.710821 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqlqn\" (UniqueName: \"kubernetes.io/projected/468b1c7e-5913-4a04-807a-8ea0a8b28506-kube-api-access-zqlqn\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:20.710975 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.710850 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd621d4c-6bc9-44bf-9af8-089078325917-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:20.712953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.712927 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd621d4c-6bc9-44bf-9af8-089078325917-kube-api-access-gb4qb" pod "cd621d4c-6bc9-44bf-9af8-089078325917" (UID: "cd621d4c-6bc9-44bf-9af8-089078325917"). InnerVolumeSpecName "kube-api-access-gb4qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:20.812218 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.812141 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4qb\" (UniqueName: \"kubernetes.io/projected/cd621d4c-6bc9-44bf-9af8-089078325917-kube-api-access-gb4qb\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:20.855528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.855501 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f"] Oct 31 00:47:20.888952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:20.888921 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f"] Oct 31 00:47:21.164303 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:21.164135 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv"] Oct 31 00:47:21.168424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:21.168394 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv"] Oct 31 00:47:21.398068 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:21.398027 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-38dd3e-759ddf7657-hlj7f" podUID="468b1c7e-5913-4a04-807a-8ea0a8b28506" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.187:50052\" within 1s: context deadline exceeded" Oct 31 00:47:21.399119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:21.399092 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38dd3e-545b56d899-xqfzv" podUID="cd621d4c-6bc9-44bf-9af8-089078325917" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.188:50052\" within 1s: context deadline exceeded" Oct 31 00:47:21.399194 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:47:21.399179 4969 logging.go:55] [core] [Channel #5560 SubChannel #5561]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.188:50052", ServerName: "10.133.0.188:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.188:50052: operation was canceled" Oct 31 00:47:21.734802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:21.734770 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468b1c7e-5913-4a04-807a-8ea0a8b28506" path="/var/lib/kubelet/pods/468b1c7e-5913-4a04-807a-8ea0a8b28506/volumes" Oct 31 00:47:21.735264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:21.735249 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd621d4c-6bc9-44bf-9af8-089078325917" path="/var/lib/kubelet/pods/cd621d4c-6bc9-44bf-9af8-089078325917/volumes" Oct 31 00:47:25.752075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.752042 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc"] Oct 31 00:47:25.752653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.752639 4969 cpu_manager.go:414] "containerMap: removing container" podUID="cd621d4c-6bc9-44bf-9af8-089078325917" containerName="payload-pre-processing" Oct 31 00:47:25.752715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.752655 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd621d4c-6bc9-44bf-9af8-089078325917" containerName="payload-pre-processing" Oct 31 00:47:25.752715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.752679 4969 cpu_manager.go:414] "containerMap: removing container" podUID="468b1c7e-5913-4a04-807a-8ea0a8b28506" containerName="payload-processing" Oct 31 00:47:25.752715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.752684 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468b1c7e-5913-4a04-807a-8ea0a8b28506" containerName="payload-processing" Oct 31 00:47:25.752817 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.752774 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="468b1c7e-5913-4a04-807a-8ea0a8b28506" containerName="payload-processing" Oct 31 00:47:25.752817 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.752788 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd621d4c-6bc9-44bf-9af8-089078325917" containerName="payload-pre-processing" Oct 31 00:47:25.757854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.757837 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.853701 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.853649 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc"] Oct 31 00:47:25.877441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.877405 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.877441 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.877450 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.877770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.877667 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.877770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.877710 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.877770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.877738 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d31630-12eb-414a-8466-64283a778ff6-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.877945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.877883 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.877945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.877931 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.878061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.877962 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5zcl\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-kube-api-access-v5zcl\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.878176 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.878159 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d31630-12eb-414a-8466-64283a778ff6-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.979870 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.979838 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d31630-12eb-414a-8466-64283a778ff6-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.979874 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.979895 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.979964 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.979984 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.980011 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d31630-12eb-414a-8466-64283a778ff6-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980324 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.980139 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980324 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.980203 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980324 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.980232 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5zcl\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-kube-api-access-v5zcl\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.980321 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.980346 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.980375 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.980583 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.980907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.980859 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d31630-12eb-414a-8466-64283a778ff6-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.982480 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.982459 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:25.982750 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:25.982732 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d31630-12eb-414a-8466-64283a778ff6-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:26.041623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.041544 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:26.041744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.041684 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5zcl\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-kube-api-access-v5zcl\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:26.072805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.072782 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:26.503943 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.503915 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc"] Oct 31 00:47:26.504945 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:47:26.504919 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d31630_12eb_414a_8466_64283a778ff6.slice/crio-f4bccbd4691c8e6b31d4c5dff68b83356585e04db6bedf65bcd09ca8f68fac53 WatchSource:0}: Error finding container f4bccbd4691c8e6b31d4c5dff68b83356585e04db6bedf65bcd09ca8f68fac53: Status 404 returned error can't find the container with id f4bccbd4691c8e6b31d4c5dff68b83356585e04db6bedf65bcd09ca8f68fac53 Oct 31 00:47:26.507467 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.507427 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:26.507581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.507512 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:26.507581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.507555 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:26.711907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.711868 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" event={"ID":"44d31630-12eb-414a-8466-64283a778ff6","Type":"ContainerStarted","Data":"65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a"} Oct 31 00:47:26.712084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.711928 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" event={"ID":"44d31630-12eb-414a-8466-64283a778ff6","Type":"ContainerStarted","Data":"f4bccbd4691c8e6b31d4c5dff68b83356585e04db6bedf65bcd09ca8f68fac53"} Oct 31 00:47:26.835175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:26.835071 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podStartSLOduration=1.8350562400000001 podStartE2EDuration="1.83505624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:26.831422162 +0000 UTC m=+2679.306780607" watchObservedRunningTime="2026-10-31 00:47:26.83505624 +0000 UTC m=+2679.310414730" Oct 31 00:47:27.072977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:27.072938 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:27.074669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:27.074643 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 31 00:47:27.074824 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:27.074691 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 31 00:47:28.074105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:28.074068 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 31 00:47:28.074489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:28.074126 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 31 00:47:29.073784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:29.073748 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 31 00:47:29.073961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:29.073798 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 31 00:47:30.073503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:30.073465 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 31 00:47:30.073940 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:30.073515 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 31 00:47:31.074103 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:31.074069 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 31 00:47:31.074524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:31.074122 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 31 00:47:32.073982 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:32.073941 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 31 00:47:32.074154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:32.074011 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 31 00:47:32.729538 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:47:32.729498 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:47:33.073414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:33.073322 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 31 00:47:33.073414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:33.073391 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 31 00:47:34.078670 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:34.078639 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:34.079183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:34.079037 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:34.079758 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:34.079736 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:47:43.831103 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:43.831068 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt"] Oct 31 00:47:43.831523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:43.831257 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" podUID="47310c95-58a8-4814-9668-f93cdc6231e6" containerName="maas-api" containerID="cri-o://dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40" gracePeriod=30 Oct 31 00:47:43.886187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:43.886151 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" podUID="47310c95-58a8-4814-9668-f93cdc6231e6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.193:8443/health\": dial tcp 10.133.0.193:8443: connect: connection refused" Oct 31 00:47:44.110277 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.110251 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:47:44.217775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.217749 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-metrics-certs\") pod \"47310c95-58a8-4814-9668-f93cdc6231e6\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " Oct 31 00:47:44.217923 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.217910 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls\") pod \"47310c95-58a8-4814-9668-f93cdc6231e6\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " Oct 31 00:47:44.218039 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.218022 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47310c95-58a8-4814-9668-f93cdc6231e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47310c95-58a8-4814-9668-f93cdc6231e6-openshift-service-ca\") pod \"47310c95-58a8-4814-9668-f93cdc6231e6\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " Oct 31 00:47:44.218136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.218120 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47310c95-58a8-4814-9668-f93cdc6231e6-kube-api-access-v76hg\" (UniqueName: \"kubernetes.io/projected/47310c95-58a8-4814-9668-f93cdc6231e6-kube-api-access-v76hg\") pod \"47310c95-58a8-4814-9668-f93cdc6231e6\" (UID: \"47310c95-58a8-4814-9668-f93cdc6231e6\") " Oct 31 00:47:44.218382 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.218353 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47310c95-58a8-4814-9668-f93cdc6231e6-openshift-service-ca" pod "47310c95-58a8-4814-9668-f93cdc6231e6" (UID: "47310c95-58a8-4814-9668-f93cdc6231e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:44.218627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.218587 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47310c95-58a8-4814-9668-f93cdc6231e6-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:44.220231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.220203 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls" pod "47310c95-58a8-4814-9668-f93cdc6231e6" (UID: "47310c95-58a8-4814-9668-f93cdc6231e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:44.220333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.220246 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-metrics-certs" pod "47310c95-58a8-4814-9668-f93cdc6231e6" (UID: "47310c95-58a8-4814-9668-f93cdc6231e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:44.220333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.220281 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47310c95-58a8-4814-9668-f93cdc6231e6-kube-api-access-v76hg" pod "47310c95-58a8-4814-9668-f93cdc6231e6" (UID: "47310c95-58a8-4814-9668-f93cdc6231e6"). InnerVolumeSpecName "kube-api-access-v76hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:44.319690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.319659 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v76hg\" (UniqueName: \"kubernetes.io/projected/47310c95-58a8-4814-9668-f93cdc6231e6-kube-api-access-v76hg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:44.319690 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.319688 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:44.319904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.319698 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47310c95-58a8-4814-9668-f93cdc6231e6-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:44.867330 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.867291 4969 generic.go:356] "Generic (PLEG): container finished" podID="47310c95-58a8-4814-9668-f93cdc6231e6" containerID="dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40" exitCode=0 Oct 31 00:47:44.867774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.867412 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" Oct 31 00:47:44.867774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.867455 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" event={"ID":"47310c95-58a8-4814-9668-f93cdc6231e6","Type":"ContainerDied","Data":"dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40"} Oct 31 00:47:44.867774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.867633 4969 scope.go:122] "RemoveContainer" containerID="dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40" Oct 31 00:47:44.867774 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.867638 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt" event={"ID":"47310c95-58a8-4814-9668-f93cdc6231e6","Type":"ContainerDied","Data":"d15b358829b7b4ff09da81c17f2e75716e54afb6f2e3cbc4612827754092026e"} Oct 31 00:47:44.936931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.936900 4969 scope.go:122] "RemoveContainer" containerID="dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40" Oct 31 00:47:44.937373 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:47:44.937341 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40\": container with ID starting with dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40 not found: ID does not exist" containerID="dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40" Oct 31 00:47:44.937495 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:44.937385 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40"} err="failed to get container status \"dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40\": rpc error: code = NotFound desc = could not find container \"dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40\": container with ID starting with dc5db49437afb9bb19eac48e7b68328fd4e88552495348ca4f900b4e3be41d40 not found: ID does not exist" Oct 31 00:47:45.051127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:45.051091 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt"] Oct 31 00:47:45.100801 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:45.100774 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-38dd3e-7c77c4688-62rdt"] Oct 31 00:47:45.735102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:45.735068 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47310c95-58a8-4814-9668-f93cdc6231e6" path="/var/lib/kubelet/pods/47310c95-58a8-4814-9668-f93cdc6231e6/volumes" Oct 31 00:47:48.206110 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:48.206081 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:47:48.218901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:48.218877 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:47:48.222343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:48.222320 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:47:48.236437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:48.236412 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:47:53.430936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:53.430902 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k"] Oct 31 00:47:53.431338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:53.431159 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" containerID="cri-o://aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc" gracePeriod=30 Oct 31 00:47:58.064794 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.064759 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:47:58.065202 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.064830 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:58.703454 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.703394 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:47:58.786984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.786948 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-credential-socket\") pod \"713e3d53-958f-4f7a-bdbb-645c326011e4\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " Oct 31 00:47:58.787150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787040 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-envoy\") pod \"713e3d53-958f-4f7a-bdbb-645c326011e4\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " Oct 31 00:47:58.787150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787091 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-certs\") pod \"713e3d53-958f-4f7a-bdbb-645c326011e4\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " Oct 31 00:47:58.787279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787154 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-podinfo\") pod \"713e3d53-958f-4f7a-bdbb-645c326011e4\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " Oct 31 00:47:58.787279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787198 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-credential-socket" pod "713e3d53-958f-4f7a-bdbb-645c326011e4" (UID: "713e3d53-958f-4f7a-bdbb-645c326011e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:58.787279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787228 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713e3d53-958f-4f7a-bdbb-645c326011e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713e3d53-958f-4f7a-bdbb-645c326011e4-istiod-ca-cert\") pod \"713e3d53-958f-4f7a-bdbb-645c326011e4\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " Oct 31 00:47:58.787279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787266 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-data\") pod \"713e3d53-958f-4f7a-bdbb-645c326011e4\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " Oct 31 00:47:58.787498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787313 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-socket\") pod \"713e3d53-958f-4f7a-bdbb-645c326011e4\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " Oct 31 00:47:58.787498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787340 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-kube-api-access-6f5l6\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-kube-api-access-6f5l6\") pod \"713e3d53-958f-4f7a-bdbb-645c326011e4\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " Oct 31 00:47:58.787498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787368 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-token\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-token\") pod \"713e3d53-958f-4f7a-bdbb-645c326011e4\" (UID: \"713e3d53-958f-4f7a-bdbb-645c326011e4\") " Oct 31 00:47:58.787498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787450 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-certs" pod "713e3d53-958f-4f7a-bdbb-645c326011e4" (UID: "713e3d53-958f-4f7a-bdbb-645c326011e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:58.787739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787618 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-socket" pod "713e3d53-958f-4f7a-bdbb-645c326011e4" (UID: "713e3d53-958f-4f7a-bdbb-645c326011e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:58.787739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787647 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-data" pod "713e3d53-958f-4f7a-bdbb-645c326011e4" (UID: "713e3d53-958f-4f7a-bdbb-645c326011e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:58.787739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787707 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713e3d53-958f-4f7a-bdbb-645c326011e4-istiod-ca-cert" pod "713e3d53-958f-4f7a-bdbb-645c326011e4" (UID: "713e3d53-958f-4f7a-bdbb-645c326011e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:58.788069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787886 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713e3d53-958f-4f7a-bdbb-645c326011e4-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.788069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787905 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.788069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787919 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.788069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787933 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.788069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.787943 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.789521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.789496 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-envoy" pod "713e3d53-958f-4f7a-bdbb-645c326011e4" (UID: "713e3d53-958f-4f7a-bdbb-645c326011e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:58.789775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.789755 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-token" pod "713e3d53-958f-4f7a-bdbb-645c326011e4" (UID: "713e3d53-958f-4f7a-bdbb-645c326011e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:58.790159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.790133 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-kube-api-access-6f5l6" pod "713e3d53-958f-4f7a-bdbb-645c326011e4" (UID: "713e3d53-958f-4f7a-bdbb-645c326011e4"). InnerVolumeSpecName "kube-api-access-6f5l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:58.790159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.790136 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-podinfo" pod "713e3d53-958f-4f7a-bdbb-645c326011e4" (UID: "713e3d53-958f-4f7a-bdbb-645c326011e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:47:58.888745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.888710 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.888745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.888742 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f5l6\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-kube-api-access-6f5l6\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.888955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.888756 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.888955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.888768 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713e3d53-958f-4f7a-bdbb-645c326011e4-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:47:58.974225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.974125 4969 generic.go:356] "Generic (PLEG): container finished" podID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerID="aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc" exitCode=0 Oct 31 00:47:58.974225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.974202 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" Oct 31 00:47:58.974225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.974209 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" event={"ID":"713e3d53-958f-4f7a-bdbb-645c326011e4","Type":"ContainerDied","Data":"aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc"} Oct 31 00:47:58.974471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.974292 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k" event={"ID":"713e3d53-958f-4f7a-bdbb-645c326011e4","Type":"ContainerDied","Data":"574bbe8eae777e059b22159f6a34ffc2ce667e9ab3ab096b28e761c2f4e15c61"} Oct 31 00:47:58.974471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.974314 4969 scope.go:122] "RemoveContainer" containerID="aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc" Oct 31 00:47:58.989382 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.989357 4969 scope.go:122] "RemoveContainer" containerID="407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166" Oct 31 00:47:58.999335 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.999316 4969 scope.go:122] "RemoveContainer" containerID="aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc" Oct 31 00:47:58.999630 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:47:58.999611 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc\": container with ID starting with aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc not found: ID does not exist" containerID="aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc" Oct 31 00:47:58.999704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.999639 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc"} err="failed to get container status \"aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc\": rpc error: code = NotFound desc = could not find container \"aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc\": container with ID starting with aa3a2e9065a037b521ac01c15cef23d66ff5ed33cf66f77dc0eb3a5a7a4ed1fc not found: ID does not exist" Oct 31 00:47:58.999704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.999656 4969 scope.go:122] "RemoveContainer" containerID="407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166" Oct 31 00:47:58.999925 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:47:58.999903 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166\": container with ID starting with 407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166 not found: ID does not exist" containerID="407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166" Oct 31 00:47:58.999970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:58.999931 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166"} err="failed to get container status \"407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166\": rpc error: code = NotFound desc = could not find container \"407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166\": container with ID starting with 407a593d754d006298463608e55e4d6b0ab20a44bf2b596e8c0ef29b3ffed166 not found: ID does not exist" Oct 31 00:47:59.140369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:59.140337 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k"] Oct 31 00:47:59.145105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:59.145082 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-38dd3e-openshift-default-65bd7fdb9c-6ph5k"] Oct 31 00:47:59.735116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:47:59.735076 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" path="/var/lib/kubelet/pods/713e3d53-958f-4f7a-bdbb-645c326011e4/volumes" Oct 31 00:48:01.938071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:01.938031 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv"] Oct 31 00:48:02.114683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:02.114643 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d"] Oct 31 00:48:02.728574 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:02.728542 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:48:06.939392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:06.939351 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" podUID="a9bb5d6b-6d38-4422-bda2-31cdeee6339f" containerName="payload-processing" containerID="cri-o://e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d" gracePeriod=25 Oct 31 00:48:07.115574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.115531 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" podUID="4a110951-261d-44ac-9f18-c091b1a27591" containerName="payload-pre-processing" containerID="cri-o://21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6" gracePeriod=25 Oct 31 00:48:07.252948 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.252920 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:48:07.380610 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.380553 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:48:07.404719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.404621 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-kube-api-access-frxsv\" (UniqueName: \"kubernetes.io/projected/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-kube-api-access-frxsv\") pod \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\" (UID: \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\") " Oct 31 00:48:07.404889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.404738 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-plugins-config-volume\") pod \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\" (UID: \"a9bb5d6b-6d38-4422-bda2-31cdeee6339f\") " Oct 31 00:48:07.405196 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.405170 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-plugins-config-volume" pod "a9bb5d6b-6d38-4422-bda2-31cdeee6339f" (UID: "a9bb5d6b-6d38-4422-bda2-31cdeee6339f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:07.405391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.405358 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:07.407223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.407197 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-kube-api-access-frxsv" pod "a9bb5d6b-6d38-4422-bda2-31cdeee6339f" (UID: "a9bb5d6b-6d38-4422-bda2-31cdeee6339f"). InnerVolumeSpecName "kube-api-access-frxsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:07.508106 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.508017 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a110951-261d-44ac-9f18-c091b1a27591-kube-api-access-p6jb8\" (UniqueName: \"kubernetes.io/projected/4a110951-261d-44ac-9f18-c091b1a27591-kube-api-access-p6jb8\") pod \"4a110951-261d-44ac-9f18-c091b1a27591\" (UID: \"4a110951-261d-44ac-9f18-c091b1a27591\") " Oct 31 00:48:07.508281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.508213 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a110951-261d-44ac-9f18-c091b1a27591-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a110951-261d-44ac-9f18-c091b1a27591-plugins-config-volume\") pod \"4a110951-261d-44ac-9f18-c091b1a27591\" (UID: \"4a110951-261d-44ac-9f18-c091b1a27591\") " Oct 31 00:48:07.508931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.508903 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxsv\" (UniqueName: \"kubernetes.io/projected/a9bb5d6b-6d38-4422-bda2-31cdeee6339f-kube-api-access-frxsv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:07.512876 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.511884 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a110951-261d-44ac-9f18-c091b1a27591-kube-api-access-p6jb8" pod "4a110951-261d-44ac-9f18-c091b1a27591" (UID: "4a110951-261d-44ac-9f18-c091b1a27591"). InnerVolumeSpecName "kube-api-access-p6jb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:07.513365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.513062 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a110951-261d-44ac-9f18-c091b1a27591-plugins-config-volume" pod "4a110951-261d-44ac-9f18-c091b1a27591" (UID: "4a110951-261d-44ac-9f18-c091b1a27591"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:07.610659 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.610621 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6jb8\" (UniqueName: \"kubernetes.io/projected/4a110951-261d-44ac-9f18-c091b1a27591-kube-api-access-p6jb8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:07.610659 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:07.610655 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a110951-261d-44ac-9f18-c091b1a27591-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:08.041503 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.041471 4969 generic.go:356] "Generic (PLEG): container finished" podID="4a110951-261d-44ac-9f18-c091b1a27591" containerID="21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6" exitCode=0 Oct 31 00:48:08.041958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.041536 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" Oct 31 00:48:08.041958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.041553 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" event={"ID":"4a110951-261d-44ac-9f18-c091b1a27591","Type":"ContainerDied","Data":"21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6"} Oct 31 00:48:08.041958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.041646 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d" event={"ID":"4a110951-261d-44ac-9f18-c091b1a27591","Type":"ContainerDied","Data":"9a8ec9a19432816f9ffd6589ca4d47b8325f122590a43fe25d0eafdde0650251"} Oct 31 00:48:08.041958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.041677 4969 scope.go:122] "RemoveContainer" containerID="21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6" Oct 31 00:48:08.043171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.043147 4969 generic.go:356] "Generic (PLEG): container finished" podID="a9bb5d6b-6d38-4422-bda2-31cdeee6339f" containerID="e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d" exitCode=0 Oct 31 00:48:08.043296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.043209 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" Oct 31 00:48:08.043422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.043337 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" event={"ID":"a9bb5d6b-6d38-4422-bda2-31cdeee6339f","Type":"ContainerDied","Data":"e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d"} Oct 31 00:48:08.043486 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.043454 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv" event={"ID":"a9bb5d6b-6d38-4422-bda2-31cdeee6339f","Type":"ContainerDied","Data":"a19e495bbc0b53bcb78a8bc12ee4cf05ce894db6dafb9cd91b1c14f25fc8817a"} Oct 31 00:48:08.052308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.052291 4969 scope.go:122] "RemoveContainer" containerID="21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6" Oct 31 00:48:08.052570 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:08.052549 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6\": container with ID starting with 21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6 not found: ID does not exist" containerID="21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6" Oct 31 00:48:08.052706 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.052584 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6"} err="failed to get container status \"21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6\": rpc error: code = NotFound desc = could not find container \"21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6\": container with ID starting with 21ab449de99f255d1ad70af22285cd885f4aea1f2599958af1c689ac471ddff6 not found: ID does not exist" Oct 31 00:48:08.052706 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.052633 4969 scope.go:122] "RemoveContainer" containerID="e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d" Oct 31 00:48:08.062010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.061994 4969 scope.go:122] "RemoveContainer" containerID="e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d" Oct 31 00:48:08.062292 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:08.062274 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d\": container with ID starting with e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d not found: ID does not exist" containerID="e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d" Oct 31 00:48:08.062341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.062298 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d"} err="failed to get container status \"e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d\": rpc error: code = NotFound desc = could not find container \"e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d\": container with ID starting with e3f0b830b351824e4fa5e8b68d7e68cb14a56490b2cdf3f04b45f7f0cd80595d not found: ID does not exist" Oct 31 00:48:08.212945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.212848 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv"] Oct 31 00:48:08.221511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.221481 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ae676-8bbc7fb4c-tfpwv"] Oct 31 00:48:08.332471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.332441 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d"] Oct 31 00:48:08.335727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:08.335707 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ae676-854c474479-jh55d"] Oct 31 00:48:09.736573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:09.736541 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a110951-261d-44ac-9f18-c091b1a27591" path="/var/lib/kubelet/pods/4a110951-261d-44ac-9f18-c091b1a27591/volumes" Oct 31 00:48:09.737035 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:09.737019 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bb5d6b-6d38-4422-bda2-31cdeee6339f" path="/var/lib/kubelet/pods/a9bb5d6b-6d38-4422-bda2-31cdeee6339f/volumes" Oct 31 00:48:26.221667 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.221634 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv"] Oct 31 00:48:26.222194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222180 4969 cpu_manager.go:414] "containerMap: removing container" podUID="47310c95-58a8-4814-9668-f93cdc6231e6" containerName="maas-api" Oct 31 00:48:26.222240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222194 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47310c95-58a8-4814-9668-f93cdc6231e6" containerName="maas-api" Oct 31 00:48:26.222240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222205 4969 cpu_manager.go:414] "containerMap: removing container" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" Oct 31 00:48:26.222240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222210 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" Oct 31 00:48:26.222240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222226 4969 cpu_manager.go:414] "containerMap: removing container" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" Oct 31 00:48:26.222240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222231 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" Oct 31 00:48:26.222397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222252 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a9bb5d6b-6d38-4422-bda2-31cdeee6339f" containerName="payload-processing" Oct 31 00:48:26.222397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222257 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bb5d6b-6d38-4422-bda2-31cdeee6339f" containerName="payload-processing" Oct 31 00:48:26.222397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222272 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4a110951-261d-44ac-9f18-c091b1a27591" containerName="payload-pre-processing" Oct 31 00:48:26.222397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222277 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a110951-261d-44ac-9f18-c091b1a27591" containerName="payload-pre-processing" Oct 31 00:48:26.222397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222361 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="47310c95-58a8-4814-9668-f93cdc6231e6" containerName="maas-api" Oct 31 00:48:26.222397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222369 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" Oct 31 00:48:26.222397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222378 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bb5d6b-6d38-4422-bda2-31cdeee6339f" containerName="payload-processing" Oct 31 00:48:26.222397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.222387 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a110951-261d-44ac-9f18-c091b1a27591" containerName="payload-pre-processing" Oct 31 00:48:26.225919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.225902 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.317214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.317182 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv"] Oct 31 00:48:26.333450 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.333419 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqnmg\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-kube-api-access-dqnmg\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.333632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.333519 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.333632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.333579 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.333789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.333668 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.333789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.333757 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.333911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.333842 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.333911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.333862 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.333911 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.333909 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d824ac7-d126-407d-b68f-10869bd7a0f4-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.334084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.333929 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.435617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.435556 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.435798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.435652 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.435798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.435752 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.435885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.435819 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.435885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.435851 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.435998 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.435906 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d824ac7-d126-407d-b68f-10869bd7a0f4-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.435998 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.435938 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.436108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.436007 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqnmg\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-kube-api-access-dqnmg\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.436108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.436018 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.436108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.436053 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.436108 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.436101 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.436322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.436210 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.436448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.436414 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.436682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.436660 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d824ac7-d126-407d-b68f-10869bd7a0f4-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.438476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.438453 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.438574 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.438554 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.496928 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.496865 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqnmg\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-kube-api-access-dqnmg\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.497050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.496972 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.540682 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.540651 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:26.975432 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.975405 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv"] Oct 31 00:48:26.976410 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:48:26.976382 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d824ac7_d126_407d_b68f_10869bd7a0f4.slice/crio-b433997365f92e56fcab774481f873aa7aed2caa3975113b1b2b66478cd45e9f WatchSource:0}: Error finding container b433997365f92e56fcab774481f873aa7aed2caa3975113b1b2b66478cd45e9f: Status 404 returned error can't find the container with id b433997365f92e56fcab774481f873aa7aed2caa3975113b1b2b66478cd45e9f Oct 31 00:48:26.978917 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.978883 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:26.979003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.978950 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:26.979003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:26.978983 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:27.185734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:27.185696 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" event={"ID":"6d824ac7-d126-407d-b68f-10869bd7a0f4","Type":"ContainerStarted","Data":"11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e"} Oct 31 00:48:27.185903 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:27.185749 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" event={"ID":"6d824ac7-d126-407d-b68f-10869bd7a0f4","Type":"ContainerStarted","Data":"b433997365f92e56fcab774481f873aa7aed2caa3975113b1b2b66478cd45e9f"} Oct 31 00:48:27.309148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:27.309039 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" podStartSLOduration=1.309025061 podStartE2EDuration="1.309025061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:27.305514644 +0000 UTC m=+2739.780873067" watchObservedRunningTime="2026-10-31 00:48:27.309025061 +0000 UTC m=+2739.784383483" Oct 31 00:48:27.540772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:27.540729 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:28.027288 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:28.027252 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td"] Oct 31 00:48:28.543415 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:28.543371 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:48:28.543864 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:28.543424 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 31 00:48:29.542012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:29.541978 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:48:29.542242 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:29.542030 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 31 00:48:30.541622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:30.541575 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:48:30.542109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:30.541640 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 31 00:48:31.006346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:31.006311 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd"] Oct 31 00:48:31.006570 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:31.006527 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" containerID="cri-o://0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105" gracePeriod=30 Oct 31 00:48:31.541568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:31.541529 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:48:31.541770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:31.541625 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 31 00:48:32.541483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:32.541439 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:48:32.541657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:32.541504 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 31 00:48:32.729982 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:32.729946 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:48:33.541125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:33.541086 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 31 00:48:33.541341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:33.541143 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 31 00:48:34.669665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:34.669625 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:34.670134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:34.670109 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:34.670887 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:34.670866 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:48:36.271433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.271369 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:48:36.272637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.272583 4969 generic.go:356] "Generic (PLEG): container finished" podID="a960f229-1a93-4df8-afd8-025576f0e464" containerID="0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105" exitCode=0 Oct 31 00:48:36.272718 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.272663 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" event={"ID":"a960f229-1a93-4df8-afd8-025576f0e464","Type":"ContainerDied","Data":"0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105"} Oct 31 00:48:36.272764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.272725 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" event={"ID":"a960f229-1a93-4df8-afd8-025576f0e464","Type":"ContainerDied","Data":"bb2a9495e2faebf98473a9b8a5fcd02b836905a8f6159bd81dca23103752286d"} Oct 31 00:48:36.272764 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.272755 4969 scope.go:122] "RemoveContainer" containerID="0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105" Oct 31 00:48:36.283755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.283733 4969 scope.go:122] "RemoveContainer" containerID="bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2" Oct 31 00:48:36.293440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.293424 4969 scope.go:122] "RemoveContainer" containerID="0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105" Oct 31 00:48:36.293721 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:36.293703 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105\": container with ID starting with 0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105 not found: ID does not exist" containerID="0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105" Oct 31 00:48:36.293803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.293727 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105"} err="failed to get container status \"0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105\": rpc error: code = NotFound desc = could not find container \"0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105\": container with ID starting with 0efb2ca0a9d0a7f6ab834d8a781eaa4224fd7489837a43960415e184409b3105 not found: ID does not exist" Oct 31 00:48:36.293803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.293746 4969 scope.go:122] "RemoveContainer" containerID="bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2" Oct 31 00:48:36.294023 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:36.294004 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2\": container with ID starting with bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2 not found: ID does not exist" containerID="bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2" Oct 31 00:48:36.294074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.294029 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2"} err="failed to get container status \"bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2\": rpc error: code = NotFound desc = could not find container \"bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2\": container with ID starting with bd9b039fc1bc597d3244660414aeef85d01e3211ed513069b03836122375afa2 not found: ID does not exist" Oct 31 00:48:36.357347 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357323 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-socket\") pod \"a960f229-1a93-4df8-afd8-025576f0e464\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " Oct 31 00:48:36.357457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357358 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-envoy\") pod \"a960f229-1a93-4df8-afd8-025576f0e464\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " Oct 31 00:48:36.357457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357401 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-certs\") pod \"a960f229-1a93-4df8-afd8-025576f0e464\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " Oct 31 00:48:36.357457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357443 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a960f229-1a93-4df8-afd8-025576f0e464-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a960f229-1a93-4df8-afd8-025576f0e464-istio-podinfo\") pod \"a960f229-1a93-4df8-afd8-025576f0e464\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " Oct 31 00:48:36.357656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357476 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-credential-socket\") pod \"a960f229-1a93-4df8-afd8-025576f0e464\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " Oct 31 00:48:36.357656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357509 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-data\") pod \"a960f229-1a93-4df8-afd8-025576f0e464\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " Oct 31 00:48:36.357656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357537 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-istio-token\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-istio-token\") pod \"a960f229-1a93-4df8-afd8-025576f0e464\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " Oct 31 00:48:36.357656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357646 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a960f229-1a93-4df8-afd8-025576f0e464-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a960f229-1a93-4df8-afd8-025576f0e464-istiod-ca-cert\") pod \"a960f229-1a93-4df8-afd8-025576f0e464\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " Oct 31 00:48:36.357862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357652 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-socket" pod "a960f229-1a93-4df8-afd8-025576f0e464" (UID: "a960f229-1a93-4df8-afd8-025576f0e464"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:36.357862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357702 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-kube-api-access-rkhsz\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-kube-api-access-rkhsz\") pod \"a960f229-1a93-4df8-afd8-025576f0e464\" (UID: \"a960f229-1a93-4df8-afd8-025576f0e464\") " Oct 31 00:48:36.357947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357878 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-certs" pod "a960f229-1a93-4df8-afd8-025576f0e464" (UID: "a960f229-1a93-4df8-afd8-025576f0e464"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:36.357947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.357900 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-credential-socket" pod "a960f229-1a93-4df8-afd8-025576f0e464" (UID: "a960f229-1a93-4df8-afd8-025576f0e464"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:36.358645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.358274 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.358645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.358298 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.358645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.358315 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.358645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.358323 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-data" pod "a960f229-1a93-4df8-afd8-025576f0e464" (UID: "a960f229-1a93-4df8-afd8-025576f0e464"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:36.358645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.358622 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a960f229-1a93-4df8-afd8-025576f0e464-istiod-ca-cert" pod "a960f229-1a93-4df8-afd8-025576f0e464" (UID: "a960f229-1a93-4df8-afd8-025576f0e464"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:36.359896 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.359872 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-envoy" pod "a960f229-1a93-4df8-afd8-025576f0e464" (UID: "a960f229-1a93-4df8-afd8-025576f0e464"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:36.360097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.360074 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a960f229-1a93-4df8-afd8-025576f0e464-istio-podinfo" pod "a960f229-1a93-4df8-afd8-025576f0e464" (UID: "a960f229-1a93-4df8-afd8-025576f0e464"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:36.360327 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.360309 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-istio-token" pod "a960f229-1a93-4df8-afd8-025576f0e464" (UID: "a960f229-1a93-4df8-afd8-025576f0e464"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:36.360512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.360498 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-kube-api-access-rkhsz" pod "a960f229-1a93-4df8-afd8-025576f0e464" (UID: "a960f229-1a93-4df8-afd8-025576f0e464"). InnerVolumeSpecName "kube-api-access-rkhsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:36.459299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.459265 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a960f229-1a93-4df8-afd8-025576f0e464-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.459299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.459293 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkhsz\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-kube-api-access-rkhsz\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.459299 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.459305 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.459734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.459313 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a960f229-1a93-4df8-afd8-025576f0e464-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.459734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.459323 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a960f229-1a93-4df8-afd8-025576f0e464-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.459734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:36.459331 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a960f229-1a93-4df8-afd8-025576f0e464-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:37.280812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:37.280782 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd" Oct 31 00:48:37.446832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:37.446800 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd"] Oct 31 00:48:37.481876 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:37.481848 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ae676-openshift-default-574fcf9f57-6g6jd"] Oct 31 00:48:37.740339 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:37.740303 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a960f229-1a93-4df8-afd8-025576f0e464" path="/var/lib/kubelet/pods/a960f229-1a93-4df8-afd8-025576f0e464/volumes" Oct 31 00:48:43.048677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.048616 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" podUID="976ab5cb-78c2-4508-8066-37627fb32aa0" containerName="main" containerID="cri-o://47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f" gracePeriod=45 Oct 31 00:48:43.325131 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.325102 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:48:43.331737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.331707 4969 generic.go:356] "Generic (PLEG): container finished" podID="976ab5cb-78c2-4508-8066-37627fb32aa0" containerID="47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f" exitCode=0 Oct 31 00:48:43.331867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.331750 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" Oct 31 00:48:43.331867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.331797 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" event={"ID":"976ab5cb-78c2-4508-8066-37627fb32aa0","Type":"ContainerDied","Data":"47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f"} Oct 31 00:48:43.331867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.331844 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td" event={"ID":"976ab5cb-78c2-4508-8066-37627fb32aa0","Type":"ContainerDied","Data":"9286708004f42e28a9282f190da356cc9edf6db4a6c6fe3df42581389abcd04d"} Oct 31 00:48:43.331867 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.331865 4969 scope.go:122] "RemoveContainer" containerID="47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f" Oct 31 00:48:43.341977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.341958 4969 scope.go:122] "RemoveContainer" containerID="47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f" Oct 31 00:48:43.342271 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:43.342250 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f\": container with ID starting with 47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f not found: ID does not exist" containerID="47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f" Oct 31 00:48:43.342364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.342280 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f"} err="failed to get container status \"47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f\": rpc error: code = NotFound desc = could not find container \"47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f\": container with ID starting with 47613782558a1e6f79989342cc24c1d7eeddcc41fad5f22f1055dff0425edc1f not found: ID does not exist" Oct 31 00:48:43.439996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.439961 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-model-cache\") pod \"976ab5cb-78c2-4508-8066-37627fb32aa0\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " Oct 31 00:48:43.439996 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.440001 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976ab5cb-78c2-4508-8066-37627fb32aa0-kube-api-access-fkf6s\" (UniqueName: \"kubernetes.io/projected/976ab5cb-78c2-4508-8066-37627fb32aa0-kube-api-access-fkf6s\") pod \"976ab5cb-78c2-4508-8066-37627fb32aa0\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " Oct 31 00:48:43.440228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.440030 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976ab5cb-78c2-4508-8066-37627fb32aa0-tls-certs\" (UniqueName: \"kubernetes.io/secret/976ab5cb-78c2-4508-8066-37627fb32aa0-tls-certs\") pod \"976ab5cb-78c2-4508-8066-37627fb32aa0\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " Oct 31 00:48:43.440228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.440096 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-home\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-home\") pod \"976ab5cb-78c2-4508-8066-37627fb32aa0\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " Oct 31 00:48:43.440228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.440161 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-dshm\") pod \"976ab5cb-78c2-4508-8066-37627fb32aa0\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " Oct 31 00:48:43.440407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.440267 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-model-cache" pod "976ab5cb-78c2-4508-8066-37627fb32aa0" (UID: "976ab5cb-78c2-4508-8066-37627fb32aa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:43.440407 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.440304 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-tmp-dir\") pod \"976ab5cb-78c2-4508-8066-37627fb32aa0\" (UID: \"976ab5cb-78c2-4508-8066-37627fb32aa0\") " Oct 31 00:48:43.440508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.440425 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-home" pod "976ab5cb-78c2-4508-8066-37627fb32aa0" (UID: "976ab5cb-78c2-4508-8066-37627fb32aa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:43.440583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.440555 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-tmp-dir" pod "976ab5cb-78c2-4508-8066-37627fb32aa0" (UID: "976ab5cb-78c2-4508-8066-37627fb32aa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:43.441046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.441025 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.441115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.441054 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.441115 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.441070 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.442535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.442510 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976ab5cb-78c2-4508-8066-37627fb32aa0-tls-certs" pod "976ab5cb-78c2-4508-8066-37627fb32aa0" (UID: "976ab5cb-78c2-4508-8066-37627fb32aa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:43.442641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.442538 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-dshm" pod "976ab5cb-78c2-4508-8066-37627fb32aa0" (UID: "976ab5cb-78c2-4508-8066-37627fb32aa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:43.442700 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.442684 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976ab5cb-78c2-4508-8066-37627fb32aa0-kube-api-access-fkf6s" pod "976ab5cb-78c2-4508-8066-37627fb32aa0" (UID: "976ab5cb-78c2-4508-8066-37627fb32aa0"). InnerVolumeSpecName "kube-api-access-fkf6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:43.498689 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.498634 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6"] Oct 31 00:48:43.498885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.498860 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" podUID="403b3bf9-5ad9-47fc-831e-d03951f9573a" containerName="maas-api" containerID="cri-o://d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658" gracePeriod=30 Oct 31 00:48:43.545649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.544087 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkf6s\" (UniqueName: \"kubernetes.io/projected/976ab5cb-78c2-4508-8066-37627fb32aa0-kube-api-access-fkf6s\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.545649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.544139 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976ab5cb-78c2-4508-8066-37627fb32aa0-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.545649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.544156 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976ab5cb-78c2-4508-8066-37627fb32aa0-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.763174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.762727 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:48:43.766216 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.766124 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td"] Oct 31 00:48:43.769733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.769710 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-87b0f8/praxis-route-87b0f8-kserve-56b97cbf9-pt6td"] Oct 31 00:48:43.847680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.847653 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403b3bf9-5ad9-47fc-831e-d03951f9573a-kube-api-access-6f649\" (UniqueName: \"kubernetes.io/projected/403b3bf9-5ad9-47fc-831e-d03951f9573a-kube-api-access-6f649\") pod \"403b3bf9-5ad9-47fc-831e-d03951f9573a\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " Oct 31 00:48:43.847919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.847748 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls\") pod \"403b3bf9-5ad9-47fc-831e-d03951f9573a\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " Oct 31 00:48:43.847919 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.847799 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-metrics-certs\") pod \"403b3bf9-5ad9-47fc-831e-d03951f9573a\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " Oct 31 00:48:43.848066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.847942 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403b3bf9-5ad9-47fc-831e-d03951f9573a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403b3bf9-5ad9-47fc-831e-d03951f9573a-openshift-service-ca\") pod \"403b3bf9-5ad9-47fc-831e-d03951f9573a\" (UID: \"403b3bf9-5ad9-47fc-831e-d03951f9573a\") " Oct 31 00:48:43.848643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.848360 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403b3bf9-5ad9-47fc-831e-d03951f9573a-openshift-service-ca" pod "403b3bf9-5ad9-47fc-831e-d03951f9573a" (UID: "403b3bf9-5ad9-47fc-831e-d03951f9573a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:43.848643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.848450 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403b3bf9-5ad9-47fc-831e-d03951f9573a-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.850446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.850418 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403b3bf9-5ad9-47fc-831e-d03951f9573a-kube-api-access-6f649" pod "403b3bf9-5ad9-47fc-831e-d03951f9573a" (UID: "403b3bf9-5ad9-47fc-831e-d03951f9573a"). InnerVolumeSpecName "kube-api-access-6f649". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:43.850660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.850645 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls" pod "403b3bf9-5ad9-47fc-831e-d03951f9573a" (UID: "403b3bf9-5ad9-47fc-831e-d03951f9573a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:43.850783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.850758 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-metrics-certs" pod "403b3bf9-5ad9-47fc-831e-d03951f9573a" (UID: "403b3bf9-5ad9-47fc-831e-d03951f9573a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:43.950334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.950239 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f649\" (UniqueName: \"kubernetes.io/projected/403b3bf9-5ad9-47fc-831e-d03951f9573a-kube-api-access-6f649\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.950334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.950281 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.950334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:43.950297 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403b3bf9-5ad9-47fc-831e-d03951f9573a-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:44.341813 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:44.341711 4969 generic.go:356] "Generic (PLEG): container finished" podID="403b3bf9-5ad9-47fc-831e-d03951f9573a" containerID="d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658" exitCode=0 Oct 31 00:48:44.341813 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:44.341761 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" Oct 31 00:48:44.341813 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:44.341785 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" event={"ID":"403b3bf9-5ad9-47fc-831e-d03951f9573a","Type":"ContainerDied","Data":"d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658"} Oct 31 00:48:44.342378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:44.341838 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6" event={"ID":"403b3bf9-5ad9-47fc-831e-d03951f9573a","Type":"ContainerDied","Data":"8080a292bbc2b3a7ce545918b1305a70b420e8d75685717f8927068347924815"} Oct 31 00:48:44.342378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:44.341865 4969 scope.go:122] "RemoveContainer" containerID="d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658" Oct 31 00:48:44.352745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:44.352726 4969 scope.go:122] "RemoveContainer" containerID="d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658" Oct 31 00:48:44.353026 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:44.353007 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658\": container with ID starting with d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658 not found: ID does not exist" containerID="d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658" Oct 31 00:48:44.353079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:44.353034 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658"} err="failed to get container status \"d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658\": rpc error: code = NotFound desc = could not find container \"d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658\": container with ID starting with d54a53b9fb01de9416114335e6f985daea14ce2d14f5bcffceb02b69e465f658 not found: ID does not exist" Oct 31 00:48:44.522178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:44.522143 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6"] Oct 31 00:48:44.570635 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:44.570586 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6a693b-866796bb46-bmdp6"] Oct 31 00:48:45.736074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:45.736039 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403b3bf9-5ad9-47fc-831e-d03951f9573a" path="/var/lib/kubelet/pods/403b3bf9-5ad9-47fc-831e-d03951f9573a/volumes" Oct 31 00:48:45.736729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:45.736709 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976ab5cb-78c2-4508-8066-37627fb32aa0" path="/var/lib/kubelet/pods/976ab5cb-78c2-4508-8066-37627fb32aa0/volumes" Oct 31 00:48:49.741472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:49.741439 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4"] Oct 31 00:48:49.741987 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:49.741644 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" podUID="cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" containerName="maas-api" containerID="cri-o://b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8" gracePeriod=30 Oct 31 00:48:50.028963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.028939 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:48:50.153208 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.153175 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-metrics-certs\") pod \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " Oct 31 00:48:50.153409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.153259 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-kube-api-access-qs6kt\" (UniqueName: \"kubernetes.io/projected/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-kube-api-access-qs6kt\") pod \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " Oct 31 00:48:50.153409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.153361 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls\") pod \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " Oct 31 00:48:50.153536 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.153409 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-openshift-service-ca\") pod \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\" (UID: \"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86\") " Oct 31 00:48:50.153889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.153861 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-openshift-service-ca" pod "cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" (UID: "cb2a48c7-089d-4ea2-b952-8c3b48d7ab86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:50.155904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.155878 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-kube-api-access-qs6kt" pod "cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" (UID: "cb2a48c7-089d-4ea2-b952-8c3b48d7ab86"). InnerVolumeSpecName "kube-api-access-qs6kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:50.156024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.155933 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls" pod "cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" (UID: "cb2a48c7-089d-4ea2-b952-8c3b48d7ab86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:50.156024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.155996 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-metrics-certs" pod "cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" (UID: "cb2a48c7-089d-4ea2-b952-8c3b48d7ab86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:50.255237 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.255207 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:50.255237 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.255237 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs6kt\" (UniqueName: \"kubernetes.io/projected/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-kube-api-access-qs6kt\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:50.255403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.255248 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:50.255403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.255258 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:50.385553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.385515 4969 generic.go:356] "Generic (PLEG): container finished" podID="cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" containerID="b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8" exitCode=0 Oct 31 00:48:50.385752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.385569 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" Oct 31 00:48:50.385752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.385566 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" event={"ID":"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86","Type":"ContainerDied","Data":"b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8"} Oct 31 00:48:50.385752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.385652 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4" event={"ID":"cb2a48c7-089d-4ea2-b952-8c3b48d7ab86","Type":"ContainerDied","Data":"506954e7f5d59ed3bcd528d190b24ffee611f3390af2ff22076487a0148d1858"} Oct 31 00:48:50.385752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.385686 4969 scope.go:122] "RemoveContainer" containerID="b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8" Oct 31 00:48:50.396837 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.396822 4969 scope.go:122] "RemoveContainer" containerID="b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8" Oct 31 00:48:50.397121 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:50.397102 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8\": container with ID starting with b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8 not found: ID does not exist" containerID="b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8" Oct 31 00:48:50.397171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.397130 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8"} err="failed to get container status \"b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8\": rpc error: code = NotFound desc = could not find container \"b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8\": container with ID starting with b75f067cfcb8813e4611b806bc0de7cfdaffe76050fe9347f6b8271c05269bd8 not found: ID does not exist" Oct 31 00:48:50.551097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.551016 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4"] Oct 31 00:48:50.600370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:50.600337 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ef8827-5d88dcd968-9n9s4"] Oct 31 00:48:51.734916 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:51.734882 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" path="/var/lib/kubelet/pods/cb2a48c7-089d-4ea2-b952-8c3b48d7ab86/volumes" Oct 31 00:48:52.647892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:52.647862 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk"] Oct 31 00:48:52.648102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:52.648081 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" containerID="cri-o://226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab" gracePeriod=30 Oct 31 00:48:56.632562 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:56.632532 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc"] Oct 31 00:48:56.632980 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:56.632762 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" containerID="cri-o://e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b" gracePeriod=30 Oct 31 00:48:57.915001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.914943 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:48:57.942771 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.942742 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-data\") pod \"5e659e33-767b-45ce-98b6-3defaa5924c1\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " Oct 31 00:48:57.942925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.942779 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-socket\") pod \"5e659e33-767b-45ce-98b6-3defaa5924c1\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " Oct 31 00:48:57.942925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.942839 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-podinfo\") pod \"5e659e33-767b-45ce-98b6-3defaa5924c1\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " Oct 31 00:48:57.942925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.942859 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-certs\") pod \"5e659e33-767b-45ce-98b6-3defaa5924c1\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " Oct 31 00:48:57.942925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.942883 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-credential-socket\") pod \"5e659e33-767b-45ce-98b6-3defaa5924c1\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " Oct 31 00:48:57.942925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.942920 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e659e33-767b-45ce-98b6-3defaa5924c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659e33-767b-45ce-98b6-3defaa5924c1-istiod-ca-cert\") pod \"5e659e33-767b-45ce-98b6-3defaa5924c1\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " Oct 31 00:48:57.943194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.942967 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-kube-api-access-8m5wn\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-kube-api-access-8m5wn\") pod \"5e659e33-767b-45ce-98b6-3defaa5924c1\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " Oct 31 00:48:57.943194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943054 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-envoy\") pod \"5e659e33-767b-45ce-98b6-3defaa5924c1\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " Oct 31 00:48:57.943194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943066 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-data" pod "5e659e33-767b-45ce-98b6-3defaa5924c1" (UID: "5e659e33-767b-45ce-98b6-3defaa5924c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:57.943194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943115 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-socket" pod "5e659e33-767b-45ce-98b6-3defaa5924c1" (UID: "5e659e33-767b-45ce-98b6-3defaa5924c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:57.943194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943153 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-token\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-token\") pod \"5e659e33-767b-45ce-98b6-3defaa5924c1\" (UID: \"5e659e33-767b-45ce-98b6-3defaa5924c1\") " Oct 31 00:48:57.943468 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943186 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-certs" pod "5e659e33-767b-45ce-98b6-3defaa5924c1" (UID: "5e659e33-767b-45ce-98b6-3defaa5924c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:57.943468 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943215 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-credential-socket" pod "5e659e33-767b-45ce-98b6-3defaa5924c1" (UID: "5e659e33-767b-45ce-98b6-3defaa5924c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:57.943752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943651 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e659e33-767b-45ce-98b6-3defaa5924c1-istiod-ca-cert" pod "5e659e33-767b-45ce-98b6-3defaa5924c1" (UID: "5e659e33-767b-45ce-98b6-3defaa5924c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:57.943752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943633 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:57.943752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943707 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:57.943752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943720 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:57.943752 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.943729 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:57.945251 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.945232 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-podinfo" pod "5e659e33-767b-45ce-98b6-3defaa5924c1" (UID: "5e659e33-767b-45ce-98b6-3defaa5924c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:57.945432 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.945416 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-token" pod "5e659e33-767b-45ce-98b6-3defaa5924c1" (UID: "5e659e33-767b-45ce-98b6-3defaa5924c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:57.945791 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.945766 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-envoy" pod "5e659e33-767b-45ce-98b6-3defaa5924c1" (UID: "5e659e33-767b-45ce-98b6-3defaa5924c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:57.945791 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:57.945781 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-kube-api-access-8m5wn" pod "5e659e33-767b-45ce-98b6-3defaa5924c1" (UID: "5e659e33-767b-45ce-98b6-3defaa5924c1"). InnerVolumeSpecName "kube-api-access-8m5wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:58.065977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.062484 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e659e33-767b-45ce-98b6-3defaa5924c1-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.065977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.062524 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5wn\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-kube-api-access-8m5wn\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.065977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.062542 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.065977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.062558 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.065977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.062573 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e659e33-767b-45ce-98b6-3defaa5924c1-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:48:58.441323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.441292 4969 generic.go:356] "Generic (PLEG): container finished" podID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerID="226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab" exitCode=0 Oct 31 00:48:58.441515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.441349 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" Oct 31 00:48:58.441515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.441382 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" event={"ID":"5e659e33-767b-45ce-98b6-3defaa5924c1","Type":"ContainerDied","Data":"226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab"} Oct 31 00:48:58.441515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.441441 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk" event={"ID":"5e659e33-767b-45ce-98b6-3defaa5924c1","Type":"ContainerDied","Data":"f18894ed73b7298092b486e5915719e4c84e146be0642aa450d2883b223e68f9"} Oct 31 00:48:58.441515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.441465 4969 scope.go:122] "RemoveContainer" containerID="226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab" Oct 31 00:48:58.453165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.453146 4969 scope.go:122] "RemoveContainer" containerID="3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4" Oct 31 00:48:58.462502 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.462486 4969 scope.go:122] "RemoveContainer" containerID="226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab" Oct 31 00:48:58.462820 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:58.462801 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab\": container with ID starting with 226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab not found: ID does not exist" containerID="226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab" Oct 31 00:48:58.462900 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.462829 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab"} err="failed to get container status \"226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab\": rpc error: code = NotFound desc = could not find container \"226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab\": container with ID starting with 226628738ace6be3594b16c9ed078ebd7b83d74d026e7dfcf72dde6246b167ab not found: ID does not exist" Oct 31 00:48:58.462900 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.462861 4969 scope.go:122] "RemoveContainer" containerID="3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4" Oct 31 00:48:58.463272 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:48:58.463104 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4\": container with ID starting with 3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4 not found: ID does not exist" containerID="3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4" Oct 31 00:48:58.463272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.463140 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4"} err="failed to get container status \"3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4\": rpc error: code = NotFound desc = could not find container \"3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4\": container with ID starting with 3a54669c4650a40632a8c2715f9bc494cd43259153a44f6ea3a697457b3ad4a4 not found: ID does not exist" Oct 31 00:48:58.607551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.607521 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk"] Oct 31 00:48:58.657956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:58.657920 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-6a693b-openshift-default-857bd997b5-vgzfk"] Oct 31 00:48:59.735251 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:48:59.735218 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" path="/var/lib/kubelet/pods/5e659e33-767b-45ce-98b6-3defaa5924c1/volumes" Oct 31 00:49:01.897316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:01.897252 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:49:02.005548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005511 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-certs\") pod \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " Oct 31 00:49:02.005745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005556 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-socket\") pod \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " Oct 31 00:49:02.005745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005611 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-token\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-token\") pod \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " Oct 31 00:49:02.005745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005640 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-data\") pod \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " Oct 31 00:49:02.005745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005691 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istiod-ca-cert\") pod \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " Oct 31 00:49:02.006007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005818 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-podinfo\") pod \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " Oct 31 00:49:02.006007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005858 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-certs" pod "4951526d-be25-4f2a-8c1a-ff8a4130ca07" (UID: "4951526d-be25-4f2a-8c1a-ff8a4130ca07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:02.006007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005883 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-credential-socket\") pod \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " Oct 31 00:49:02.006007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005893 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-socket" pod "4951526d-be25-4f2a-8c1a-ff8a4130ca07" (UID: "4951526d-be25-4f2a-8c1a-ff8a4130ca07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:02.006007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005919 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-kube-api-access-7lj86\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-kube-api-access-7lj86\") pod \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " Oct 31 00:49:02.006007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.005991 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-envoy\") pod \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\" (UID: \"4951526d-be25-4f2a-8c1a-ff8a4130ca07\") " Oct 31 00:49:02.006346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.006012 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-data" pod "4951526d-be25-4f2a-8c1a-ff8a4130ca07" (UID: "4951526d-be25-4f2a-8c1a-ff8a4130ca07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:02.006346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.006111 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istiod-ca-cert" pod "4951526d-be25-4f2a-8c1a-ff8a4130ca07" (UID: "4951526d-be25-4f2a-8c1a-ff8a4130ca07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:02.006346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.006126 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-credential-socket" pod "4951526d-be25-4f2a-8c1a-ff8a4130ca07" (UID: "4951526d-be25-4f2a-8c1a-ff8a4130ca07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:02.006473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.006450 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.006514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.006471 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.006514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.006488 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.006514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.006503 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.006669 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.006577 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.008274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.008249 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-token" pod "4951526d-be25-4f2a-8c1a-ff8a4130ca07" (UID: "4951526d-be25-4f2a-8c1a-ff8a4130ca07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:02.008401 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.008268 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-kube-api-access-7lj86" pod "4951526d-be25-4f2a-8c1a-ff8a4130ca07" (UID: "4951526d-be25-4f2a-8c1a-ff8a4130ca07"). InnerVolumeSpecName "kube-api-access-7lj86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:02.008639 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.008618 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-envoy" pod "4951526d-be25-4f2a-8c1a-ff8a4130ca07" (UID: "4951526d-be25-4f2a-8c1a-ff8a4130ca07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:02.008702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.008636 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-podinfo" pod "4951526d-be25-4f2a-8c1a-ff8a4130ca07" (UID: "4951526d-be25-4f2a-8c1a-ff8a4130ca07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:02.108174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.108075 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.108174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.108105 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lj86\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-kube-api-access-7lj86\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.108174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.108115 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.108174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.108123 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4951526d-be25-4f2a-8c1a-ff8a4130ca07-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.471645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.471585 4969 generic.go:356] "Generic (PLEG): container finished" podID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerID="e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b" exitCode=0 Oct 31 00:49:02.471645 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.471639 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" event={"ID":"4951526d-be25-4f2a-8c1a-ff8a4130ca07","Type":"ContainerDied","Data":"e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b"} Oct 31 00:49:02.471865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.471666 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" Oct 31 00:49:02.471865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.471719 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc" event={"ID":"4951526d-be25-4f2a-8c1a-ff8a4130ca07","Type":"ContainerDied","Data":"a28555e08b48f911b6b852efb505d7957acb3628d693d264e67d875ccf7f354f"} Oct 31 00:49:02.471865 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.471741 4969 scope.go:122] "RemoveContainer" containerID="e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b" Oct 31 00:49:02.482779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.482751 4969 scope.go:122] "RemoveContainer" containerID="e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b" Oct 31 00:49:02.483059 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:49:02.483035 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b\": container with ID starting with e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b not found: ID does not exist" containerID="e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b" Oct 31 00:49:02.483216 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.483071 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b"} err="failed to get container status \"e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b\": rpc error: code = NotFound desc = could not find container \"e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b\": container with ID starting with e654dcd9296532675e5e5d6400085ef49f36b82729c0bf4a0986c73c5bc59d8b not found: ID does not exist" Oct 31 00:49:02.638863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.638823 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc"] Oct 31 00:49:02.642459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:02.642426 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-kf9dc"] Oct 31 00:49:02.728731 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:49:02.728645 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:49:03.734812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:03.734782 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" path="/var/lib/kubelet/pods/4951526d-be25-4f2a-8c1a-ff8a4130ca07/volumes" Oct 31 00:49:10.099010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.098974 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252"] Oct 31 00:49:10.099492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099478 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" Oct 31 00:49:10.099540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099493 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" Oct 31 00:49:10.099540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099512 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" Oct 31 00:49:10.099540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099517 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" Oct 31 00:49:10.099540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099528 4969 cpu_manager.go:414] "containerMap: removing container" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" Oct 31 00:49:10.099540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099533 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" Oct 31 00:49:10.099540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099540 4969 cpu_manager.go:414] "containerMap: removing container" podUID="403b3bf9-5ad9-47fc-831e-d03951f9573a" containerName="maas-api" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099544 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403b3bf9-5ad9-47fc-831e-d03951f9573a" containerName="maas-api" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099553 4969 cpu_manager.go:414] "containerMap: removing container" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099557 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099578 4969 cpu_manager.go:414] "containerMap: removing container" podUID="cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" containerName="maas-api" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099583 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" containerName="maas-api" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099611 4969 cpu_manager.go:414] "containerMap: removing container" podUID="976ab5cb-78c2-4508-8066-37627fb32aa0" containerName="main" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099616 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976ab5cb-78c2-4508-8066-37627fb32aa0" containerName="main" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099704 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099712 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2a48c7-089d-4ea2-b952-8c3b48d7ab86" containerName="maas-api" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099720 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="976ab5cb-78c2-4508-8066-37627fb32aa0" containerName="main" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099728 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4951526d-be25-4f2a-8c1a-ff8a4130ca07" containerName="istio-proxy" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099734 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="713e3d53-958f-4f7a-bdbb-645c326011e4" containerName="istio-proxy" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099743 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099749 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="403b3bf9-5ad9-47fc-831e-d03951f9573a" containerName="maas-api" Oct 31 00:49:10.099751 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.099756 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e659e33-767b-45ce-98b6-3defaa5924c1" containerName="istio-proxy" Oct 31 00:49:10.104570 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.104550 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.160992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.160964 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252"] Oct 31 00:49:10.206145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.206113 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:10.206287 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.206149 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:49:10.239984 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.239954 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91841e0d-sn252\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.240124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.240018 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl5wb\" (UniqueName: \"kubernetes.io/projected/5fe6f9a9-967e-4401-872b-b510e15591e2-kube-api-access-vl5wb\") pod \"maas-api-revoke-sub-91841e0d-sn252\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.240252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.240229 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91841e0d-sn252\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.340970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.340942 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91841e0d-sn252\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.341121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.341005 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91841e0d-sn252\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.341121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.341056 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5wb\" (UniqueName: \"kubernetes.io/projected/5fe6f9a9-967e-4401-872b-b510e15591e2-kube-api-access-vl5wb\") pod \"maas-api-revoke-sub-91841e0d-sn252\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.341673 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.341653 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91841e0d-sn252\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.343394 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.343374 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91841e0d-sn252\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.402744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.402721 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5wb\" (UniqueName: \"kubernetes.io/projected/5fe6f9a9-967e-4401-872b-b510e15591e2-kube-api-access-vl5wb\") pod \"maas-api-revoke-sub-91841e0d-sn252\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.417522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.417503 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:10.795178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:10.795148 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252"] Oct 31 00:49:10.796324 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:49:10.796299 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe6f9a9_967e_4401_872b_b510e15591e2.slice/crio-027e8af7e846bc8f599b54038784da4703bcf34dd7268f556515798901656fcf WatchSource:0}: Error finding container 027e8af7e846bc8f599b54038784da4703bcf34dd7268f556515798901656fcf: Status 404 returned error can't find the container with id 027e8af7e846bc8f599b54038784da4703bcf34dd7268f556515798901656fcf Oct 31 00:49:11.544132 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:11.544095 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" event={"ID":"5fe6f9a9-967e-4401-872b-b510e15591e2","Type":"ContainerStarted","Data":"b847785d7d32edc276d7fc70d99c28a199a979126e263afd25e897075b37bfba"} Oct 31 00:49:11.544528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:11.544180 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" event={"ID":"5fe6f9a9-967e-4401-872b-b510e15591e2","Type":"ContainerStarted","Data":"027e8af7e846bc8f599b54038784da4703bcf34dd7268f556515798901656fcf"} Oct 31 00:49:11.628529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:11.628495 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b"] Oct 31 00:49:11.628730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:11.628710 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" containerID="cri-o://91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710" gracePeriod=30 Oct 31 00:49:11.688720 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:11.688660 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" podStartSLOduration=1.688641099 podStartE2EDuration="1.688641099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:11.68787505 +0000 UTC m=+2784.163233472" watchObservedRunningTime="2026-10-31 00:49:11.688641099 +0000 UTC m=+2784.163999527" Oct 31 00:49:12.555221 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:12.555185 4969 generic.go:356] "Generic (PLEG): container finished" podID="5fe6f9a9-967e-4401-872b-b510e15591e2" containerID="b847785d7d32edc276d7fc70d99c28a199a979126e263afd25e897075b37bfba" exitCode=0 Oct 31 00:49:12.555709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:12.555222 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" event={"ID":"5fe6f9a9-967e-4401-872b-b510e15591e2","Type":"ContainerDied","Data":"b847785d7d32edc276d7fc70d99c28a199a979126e263afd25e897075b37bfba"} Oct 31 00:49:13.710732 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.710710 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:13.781618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.781567 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-openshift-ca\") pod \"5fe6f9a9-967e-4401-872b-b510e15591e2\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " Oct 31 00:49:13.781762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.781655 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe6f9a9-967e-4401-872b-b510e15591e2-kube-api-access-vl5wb\" (UniqueName: \"kubernetes.io/projected/5fe6f9a9-967e-4401-872b-b510e15591e2-kube-api-access-vl5wb\") pod \"5fe6f9a9-967e-4401-872b-b510e15591e2\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " Oct 31 00:49:13.781762 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.781709 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-xks-ca\") pod \"5fe6f9a9-967e-4401-872b-b510e15591e2\" (UID: \"5fe6f9a9-967e-4401-872b-b510e15591e2\") " Oct 31 00:49:13.781890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.781868 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-openshift-ca" pod "5fe6f9a9-967e-4401-872b-b510e15591e2" (UID: "5fe6f9a9-967e-4401-872b-b510e15591e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:13.782419 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.782395 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.784502 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.784474 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-xks-ca" pod "5fe6f9a9-967e-4401-872b-b510e15591e2" (UID: "5fe6f9a9-967e-4401-872b-b510e15591e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:13.784636 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.784505 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe6f9a9-967e-4401-872b-b510e15591e2-kube-api-access-vl5wb" pod "5fe6f9a9-967e-4401-872b-b510e15591e2" (UID: "5fe6f9a9-967e-4401-872b-b510e15591e2"). InnerVolumeSpecName "kube-api-access-vl5wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:13.883020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.882990 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl5wb\" (UniqueName: \"kubernetes.io/projected/5fe6f9a9-967e-4401-872b-b510e15591e2-kube-api-access-vl5wb\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.883020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:13.883019 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe6f9a9-967e-4401-872b-b510e15591e2-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:14.570190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:14.570160 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" Oct 31 00:49:14.570365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:14.570208 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252" event={"ID":"5fe6f9a9-967e-4401-872b-b510e15591e2","Type":"ContainerDied","Data":"027e8af7e846bc8f599b54038784da4703bcf34dd7268f556515798901656fcf"} Oct 31 00:49:14.570365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:14.570261 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027e8af7e846bc8f599b54038784da4703bcf34dd7268f556515798901656fcf" Oct 31 00:49:16.061467 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.061431 4969 patch_prober.go:29] interesting pod/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:49:16.061910 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.061490 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:16.902712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.902658 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:49:16.917267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917244 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-kube-api-access-mn22m\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-kube-api-access-mn22m\") pod \"16f0772e-b63a-4552-8483-98f88c7a3fd7\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " Oct 31 00:49:16.917373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917358 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-token\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-token\") pod \"16f0772e-b63a-4552-8483-98f88c7a3fd7\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " Oct 31 00:49:16.917415 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917396 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-podinfo\") pod \"16f0772e-b63a-4552-8483-98f88c7a3fd7\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " Oct 31 00:49:16.917472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917436 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f0772e-b63a-4552-8483-98f88c7a3fd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f0772e-b63a-4552-8483-98f88c7a3fd7-istiod-ca-cert\") pod \"16f0772e-b63a-4552-8483-98f88c7a3fd7\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " Oct 31 00:49:16.917532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917470 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-socket\") pod \"16f0772e-b63a-4552-8483-98f88c7a3fd7\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " Oct 31 00:49:16.917532 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917508 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-envoy\") pod \"16f0772e-b63a-4552-8483-98f88c7a3fd7\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " Oct 31 00:49:16.917691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917543 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-data\") pod \"16f0772e-b63a-4552-8483-98f88c7a3fd7\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " Oct 31 00:49:16.917691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917580 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-credential-socket\") pod \"16f0772e-b63a-4552-8483-98f88c7a3fd7\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " Oct 31 00:49:16.917691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917655 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-certs\") pod \"16f0772e-b63a-4552-8483-98f88c7a3fd7\" (UID: \"16f0772e-b63a-4552-8483-98f88c7a3fd7\") " Oct 31 00:49:16.917967 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.917938 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-data" pod "16f0772e-b63a-4552-8483-98f88c7a3fd7" (UID: "16f0772e-b63a-4552-8483-98f88c7a3fd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:16.918266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.918202 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-socket" pod "16f0772e-b63a-4552-8483-98f88c7a3fd7" (UID: "16f0772e-b63a-4552-8483-98f88c7a3fd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:16.918266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.918245 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f0772e-b63a-4552-8483-98f88c7a3fd7-istiod-ca-cert" pod "16f0772e-b63a-4552-8483-98f88c7a3fd7" (UID: "16f0772e-b63a-4552-8483-98f88c7a3fd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:16.918395 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.918306 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-credential-socket" pod "16f0772e-b63a-4552-8483-98f88c7a3fd7" (UID: "16f0772e-b63a-4552-8483-98f88c7a3fd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:16.918459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.918425 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-certs" pod "16f0772e-b63a-4552-8483-98f88c7a3fd7" (UID: "16f0772e-b63a-4552-8483-98f88c7a3fd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:16.918522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.918500 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f0772e-b63a-4552-8483-98f88c7a3fd7-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:16.918579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.918520 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:16.918579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.918533 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:16.918579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.918548 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:16.919737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.919712 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-kube-api-access-mn22m" pod "16f0772e-b63a-4552-8483-98f88c7a3fd7" (UID: "16f0772e-b63a-4552-8483-98f88c7a3fd7"). InnerVolumeSpecName "kube-api-access-mn22m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:16.919855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.919839 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-podinfo" pod "16f0772e-b63a-4552-8483-98f88c7a3fd7" (UID: "16f0772e-b63a-4552-8483-98f88c7a3fd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:16.920200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.920181 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-token" pod "16f0772e-b63a-4552-8483-98f88c7a3fd7" (UID: "16f0772e-b63a-4552-8483-98f88c7a3fd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:16.920377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:16.920357 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-envoy" pod "16f0772e-b63a-4552-8483-98f88c7a3fd7" (UID: "16f0772e-b63a-4552-8483-98f88c7a3fd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:17.020121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.020089 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.020121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.020118 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.020121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.020128 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.020350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.020136 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f0772e-b63a-4552-8483-98f88c7a3fd7-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.020350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.020146 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn22m\" (UniqueName: \"kubernetes.io/projected/16f0772e-b63a-4552-8483-98f88c7a3fd7-kube-api-access-mn22m\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:17.592541 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.592500 4969 generic.go:356] "Generic (PLEG): container finished" podID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerID="91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710" exitCode=0 Oct 31 00:49:17.593018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.592549 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" Oct 31 00:49:17.593018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.592571 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" event={"ID":"16f0772e-b63a-4552-8483-98f88c7a3fd7","Type":"ContainerDied","Data":"91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710"} Oct 31 00:49:17.593018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.592659 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b" event={"ID":"16f0772e-b63a-4552-8483-98f88c7a3fd7","Type":"ContainerDied","Data":"1195dc36cd97fc3c792b018d8a468e19d3b4cb1a79d96e923e5c2ba07f098619"} Oct 31 00:49:17.593018 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.592682 4969 scope.go:122] "RemoveContainer" containerID="91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710" Oct 31 00:49:17.604653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.604637 4969 scope.go:122] "RemoveContainer" containerID="8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790" Oct 31 00:49:17.614038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.613929 4969 scope.go:122] "RemoveContainer" containerID="91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710" Oct 31 00:49:17.614232 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:49:17.614203 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710\": container with ID starting with 91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710 not found: ID does not exist" containerID="91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710" Oct 31 00:49:17.614314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.614245 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710"} err="failed to get container status \"91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710\": rpc error: code = NotFound desc = could not find container \"91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710\": container with ID starting with 91a3de438f690ce713cc87533ef348d803429a0ad61e2c1951cfe260d11e5710 not found: ID does not exist" Oct 31 00:49:17.614314 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.614268 4969 scope.go:122] "RemoveContainer" containerID="8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790" Oct 31 00:49:17.614567 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:49:17.614549 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790\": container with ID starting with 8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790 not found: ID does not exist" containerID="8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790" Oct 31 00:49:17.614653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.614575 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790"} err="failed to get container status \"8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790\": rpc error: code = NotFound desc = could not find container \"8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790\": container with ID starting with 8987884f6ec04a0afece45d940f07595900f30b94974773d9211778636221790 not found: ID does not exist" Oct 31 00:49:17.809137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.809104 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b"] Oct 31 00:49:17.858857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:17.858771 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-87b0f8-openshift-default-75b85b8965-kzm5b"] Oct 31 00:49:19.735109 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:19.735074 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" path="/var/lib/kubelet/pods/16f0772e-b63a-4552-8483-98f88c7a3fd7/volumes" Oct 31 00:49:20.553079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.553031 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl"] Oct 31 00:49:20.553375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.553346 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" podUID="6a5d428a-0d78-49c6-89f3-62fa31adb3cc" containerName="maas-api" containerID="cri-o://fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90" gracePeriod=30 Oct 31 00:49:20.617393 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.617360 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j"] Oct 31 00:49:20.617885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.617872 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" Oct 31 00:49:20.617941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.617886 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" Oct 31 00:49:20.617941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.617911 4969 cpu_manager.go:414] "containerMap: removing container" podUID="5fe6f9a9-967e-4401-872b-b510e15591e2" containerName="revoke-keys" Oct 31 00:49:20.617941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.617919 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe6f9a9-967e-4401-872b-b510e15591e2" containerName="revoke-keys" Oct 31 00:49:20.618046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.617942 4969 cpu_manager.go:414] "containerMap: removing container" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" Oct 31 00:49:20.618046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.617947 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" Oct 31 00:49:20.618046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.617968 4969 cpu_manager.go:414] "containerMap: removing container" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" Oct 31 00:49:20.618046 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.617974 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" Oct 31 00:49:20.618164 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.618054 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" Oct 31 00:49:20.618164 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.618066 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe6f9a9-967e-4401-872b-b510e15591e2" containerName="revoke-keys" Oct 31 00:49:20.618164 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.618076 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="a960f229-1a93-4df8-afd8-025576f0e464" containerName="istio-proxy" Oct 31 00:49:20.621777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.621757 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.660215 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.660179 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd9dcfb9-smz9j\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.660610 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.660277 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd9dcfb9-smz9j\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.660610 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.660356 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zngm\" (UniqueName: \"kubernetes.io/projected/443f3255-5bd9-4902-83fd-47a3bea32316-kube-api-access-7zngm\") pod \"maas-api-revoke-sub-bd9dcfb9-smz9j\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.675036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.675015 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:20.675296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.675274 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:49:20.680978 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.680946 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j"] Oct 31 00:49:20.761947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.761912 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd9dcfb9-smz9j\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.762434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.761975 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd9dcfb9-smz9j\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.762434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.762173 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zngm\" (UniqueName: \"kubernetes.io/projected/443f3255-5bd9-4902-83fd-47a3bea32316-kube-api-access-7zngm\") pod \"maas-api-revoke-sub-bd9dcfb9-smz9j\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.762848 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.762824 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd9dcfb9-smz9j\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.765163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.765136 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd9dcfb9-smz9j\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.825976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.825941 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zngm\" (UniqueName: \"kubernetes.io/projected/443f3255-5bd9-4902-83fd-47a3bea32316-kube-api-access-7zngm\") pod \"maas-api-revoke-sub-bd9dcfb9-smz9j\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:20.849564 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.849537 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:49:20.965301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.965270 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-kube-api-access-ggc2h\" (UniqueName: \"kubernetes.io/projected/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-kube-api-access-ggc2h\") pod \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " Oct 31 00:49:20.965501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.965332 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-metrics-certs\") pod \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " Oct 31 00:49:20.965501 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.965466 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls\") pod \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " Oct 31 00:49:20.965651 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.965502 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-openshift-service-ca\") pod \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\" (UID: \"6a5d428a-0d78-49c6-89f3-62fa31adb3cc\") " Oct 31 00:49:20.965921 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.965896 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-openshift-service-ca" pod "6a5d428a-0d78-49c6-89f3-62fa31adb3cc" (UID: "6a5d428a-0d78-49c6-89f3-62fa31adb3cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:20.968357 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.968329 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-kube-api-access-ggc2h" pod "6a5d428a-0d78-49c6-89f3-62fa31adb3cc" (UID: "6a5d428a-0d78-49c6-89f3-62fa31adb3cc"). InnerVolumeSpecName "kube-api-access-ggc2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:20.968487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.968387 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-metrics-certs" pod "6a5d428a-0d78-49c6-89f3-62fa31adb3cc" (UID: "6a5d428a-0d78-49c6-89f3-62fa31adb3cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:20.968487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.968405 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls" pod "6a5d428a-0d78-49c6-89f3-62fa31adb3cc" (UID: "6a5d428a-0d78-49c6-89f3-62fa31adb3cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:20.970633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:20.970606 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:21.067014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.066925 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggc2h\" (UniqueName: \"kubernetes.io/projected/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-kube-api-access-ggc2h\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:21.067014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.066960 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:21.067014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.066975 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:21.067014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.066987 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5d428a-0d78-49c6-89f3-62fa31adb3cc-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:21.351930 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.351902 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j"] Oct 31 00:49:21.352789 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:49:21.352766 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443f3255_5bd9_4902_83fd_47a3bea32316.slice/crio-1f3af6fdd7091b850e0ce61dc3553313b6c50f535214b4f6561491dac325e8b0 WatchSource:0}: Error finding container 1f3af6fdd7091b850e0ce61dc3553313b6c50f535214b4f6561491dac325e8b0: Status 404 returned error can't find the container with id 1f3af6fdd7091b850e0ce61dc3553313b6c50f535214b4f6561491dac325e8b0 Oct 31 00:49:21.628071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.628033 4969 generic.go:356] "Generic (PLEG): container finished" podID="6a5d428a-0d78-49c6-89f3-62fa31adb3cc" containerID="fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90" exitCode=0 Oct 31 00:49:21.628260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.628108 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" Oct 31 00:49:21.628260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.628195 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" event={"ID":"6a5d428a-0d78-49c6-89f3-62fa31adb3cc","Type":"ContainerDied","Data":"fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90"} Oct 31 00:49:21.628260 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.628246 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl" event={"ID":"6a5d428a-0d78-49c6-89f3-62fa31adb3cc","Type":"ContainerDied","Data":"d496695cb6f85735809ee1c35b91c034732ea1328700d8d46c77ab989c7c7ba8"} Oct 31 00:49:21.628439 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.628276 4969 scope.go:122] "RemoveContainer" containerID="fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90" Oct 31 00:49:21.630978 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.630950 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" event={"ID":"443f3255-5bd9-4902-83fd-47a3bea32316","Type":"ContainerStarted","Data":"1f3af6fdd7091b850e0ce61dc3553313b6c50f535214b4f6561491dac325e8b0"} Oct 31 00:49:21.650475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.649776 4969 scope.go:122] "RemoveContainer" containerID="fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90" Oct 31 00:49:21.650475 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:49:21.650222 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90\": container with ID starting with fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90 not found: ID does not exist" containerID="fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90" Oct 31 00:49:21.650475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.650253 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90"} err="failed to get container status \"fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90\": rpc error: code = NotFound desc = could not find container \"fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90\": container with ID starting with fa7d6ca7666aafc201a2d632d27e256ae4e074c1709f44c647117a682d655e90 not found: ID does not exist" Oct 31 00:49:21.795341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.795309 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl"] Oct 31 00:49:21.844668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:21.844628 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31144e-5ff9d4484b-r48sl"] Oct 31 00:49:22.640577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:22.640543 4969 generic.go:356] "Generic (PLEG): container finished" podID="443f3255-5bd9-4902-83fd-47a3bea32316" containerID="7d7994965e6398c6bc2d4ed31d98d4771eee5cf311bee1342c38f25de400b2ca" exitCode=0 Oct 31 00:49:22.640781 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:22.640639 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" event={"ID":"443f3255-5bd9-4902-83fd-47a3bea32316","Type":"ContainerDied","Data":"7d7994965e6398c6bc2d4ed31d98d4771eee5cf311bee1342c38f25de400b2ca"} Oct 31 00:49:23.735420 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:23.735383 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5d428a-0d78-49c6-89f3-62fa31adb3cc" path="/var/lib/kubelet/pods/6a5d428a-0d78-49c6-89f3-62fa31adb3cc/volumes" Oct 31 00:49:23.804726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:23.804699 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:23.900152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:23.900062 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-xks-ca\") pod \"443f3255-5bd9-4902-83fd-47a3bea32316\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " Oct 31 00:49:23.900316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:23.900202 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443f3255-5bd9-4902-83fd-47a3bea32316-kube-api-access-7zngm\" (UniqueName: \"kubernetes.io/projected/443f3255-5bd9-4902-83fd-47a3bea32316-kube-api-access-7zngm\") pod \"443f3255-5bd9-4902-83fd-47a3bea32316\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " Oct 31 00:49:23.900399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:23.900352 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-openshift-ca\") pod \"443f3255-5bd9-4902-83fd-47a3bea32316\" (UID: \"443f3255-5bd9-4902-83fd-47a3bea32316\") " Oct 31 00:49:23.900736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:23.900712 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-openshift-ca" pod "443f3255-5bd9-4902-83fd-47a3bea32316" (UID: "443f3255-5bd9-4902-83fd-47a3bea32316"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:23.902412 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:23.902393 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443f3255-5bd9-4902-83fd-47a3bea32316-kube-api-access-7zngm" pod "443f3255-5bd9-4902-83fd-47a3bea32316" (UID: "443f3255-5bd9-4902-83fd-47a3bea32316"). InnerVolumeSpecName "kube-api-access-7zngm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:23.902483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:23.902446 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-xks-ca" pod "443f3255-5bd9-4902-83fd-47a3bea32316" (UID: "443f3255-5bd9-4902-83fd-47a3bea32316"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:24.001950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:24.001913 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zngm\" (UniqueName: \"kubernetes.io/projected/443f3255-5bd9-4902-83fd-47a3bea32316-kube-api-access-7zngm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:24.001950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:24.001943 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:24.001950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:24.001954 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443f3255-5bd9-4902-83fd-47a3bea32316-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:24.656298 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:24.656267 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" Oct 31 00:49:24.656491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:24.656302 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j" event={"ID":"443f3255-5bd9-4902-83fd-47a3bea32316","Type":"ContainerDied","Data":"1f3af6fdd7091b850e0ce61dc3553313b6c50f535214b4f6561491dac325e8b0"} Oct 31 00:49:24.656491 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:24.656387 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f3af6fdd7091b850e0ce61dc3553313b6c50f535214b4f6561491dac325e8b0" Oct 31 00:49:32.770623 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:49:32.770555 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:49:35.086515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.086485 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2"] Oct 31 00:49:35.087003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.086989 4969 cpu_manager.go:414] "containerMap: removing container" podUID="6a5d428a-0d78-49c6-89f3-62fa31adb3cc" containerName="maas-api" Oct 31 00:49:35.087049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.087003 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5d428a-0d78-49c6-89f3-62fa31adb3cc" containerName="maas-api" Oct 31 00:49:35.087049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.087012 4969 cpu_manager.go:414] "containerMap: removing container" podUID="443f3255-5bd9-4902-83fd-47a3bea32316" containerName="revoke-keys" Oct 31 00:49:35.087049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.087017 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443f3255-5bd9-4902-83fd-47a3bea32316" containerName="revoke-keys" Oct 31 00:49:35.087154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.087121 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f0772e-b63a-4552-8483-98f88c7a3fd7" containerName="istio-proxy" Oct 31 00:49:35.087154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.087130 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5d428a-0d78-49c6-89f3-62fa31adb3cc" containerName="maas-api" Oct 31 00:49:35.087154 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.087136 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="443f3255-5bd9-4902-83fd-47a3bea32316" containerName="revoke-keys" Oct 31 00:49:35.093278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.093249 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.148800 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.148763 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2"] Oct 31 00:49:35.194572 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.194543 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:49:35.194769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.194577 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:35.228781 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.228746 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj"] Oct 31 00:49:35.232709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.232690 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.250279 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.250255 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32d3d330-l7gk2\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.250381 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.250300 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32d3d330-l7gk2\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.250440 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.250429 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28dk\" (UniqueName: \"kubernetes.io/projected/531a267e-a00f-4791-b9df-bcd217f4368c-kube-api-access-c28dk\") pod \"maas-api-revoke-sub-32d3d330-l7gk2\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.290773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.290741 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj"] Oct 31 00:49:35.352117 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.352043 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32d3d330-l7gk2\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.352117 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.352093 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32d3d330-l7gk2\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.352305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.352178 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0db33ab-knsmj\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.352305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.352211 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28dk\" (UniqueName: \"kubernetes.io/projected/531a267e-a00f-4791-b9df-bcd217f4368c-kube-api-access-c28dk\") pod \"maas-api-revoke-sub-32d3d330-l7gk2\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.352305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.352237 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57n5b\" (UniqueName: \"kubernetes.io/projected/13c148b9-60f6-4df0-a473-dd731078121d-kube-api-access-57n5b\") pod \"maas-api-revoke-sub-c0db33ab-knsmj\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.352305 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.352257 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0db33ab-knsmj\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.352823 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.352799 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32d3d330-l7gk2\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.354643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.354622 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32d3d330-l7gk2\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.415398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.415369 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28dk\" (UniqueName: \"kubernetes.io/projected/531a267e-a00f-4791-b9df-bcd217f4368c-kube-api-access-c28dk\") pod \"maas-api-revoke-sub-32d3d330-l7gk2\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.453128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.453102 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0db33ab-knsmj\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.453263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.453170 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57n5b\" (UniqueName: \"kubernetes.io/projected/13c148b9-60f6-4df0-a473-dd731078121d-kube-api-access-57n5b\") pod \"maas-api-revoke-sub-c0db33ab-knsmj\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.453263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.453203 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0db33ab-knsmj\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.453905 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.453882 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0db33ab-knsmj\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.455457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.455432 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0db33ab-knsmj\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.514262 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.514236 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57n5b\" (UniqueName: \"kubernetes.io/projected/13c148b9-60f6-4df0-a473-dd731078121d-kube-api-access-57n5b\") pod \"maas-api-revoke-sub-c0db33ab-knsmj\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.549063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.549041 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:35.708459 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.708425 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:35.934677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:35.934652 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj"] Oct 31 00:49:35.935612 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:49:35.935568 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c148b9_60f6_4df0_a473_dd731078121d.slice/crio-bce749cb48f22b52d049da57c53253d5b1ee63e970c7301b9ee2c0a710a9c036 WatchSource:0}: Error finding container bce749cb48f22b52d049da57c53253d5b1ee63e970c7301b9ee2c0a710a9c036: Status 404 returned error can't find the container with id bce749cb48f22b52d049da57c53253d5b1ee63e970c7301b9ee2c0a710a9c036 Oct 31 00:49:36.088308 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:36.088286 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2"] Oct 31 00:49:36.091058 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:49:36.091023 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531a267e_a00f_4791_b9df_bcd217f4368c.slice/crio-aa1a985ee96267d742d8dc091e2d0c3c3cb08ec55b1f7d1dd44cfbe8f42ff242 WatchSource:0}: Error finding container aa1a985ee96267d742d8dc091e2d0c3c3cb08ec55b1f7d1dd44cfbe8f42ff242: Status 404 returned error can't find the container with id aa1a985ee96267d742d8dc091e2d0c3c3cb08ec55b1f7d1dd44cfbe8f42ff242 Oct 31 00:49:36.784172 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:36.784135 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" event={"ID":"531a267e-a00f-4791-b9df-bcd217f4368c","Type":"ContainerStarted","Data":"9f5d1b10b92d33939b2eca44852c52d95a7961058d34aa42f96ca78acda557c5"} Oct 31 00:49:36.784334 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:36.784199 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" event={"ID":"531a267e-a00f-4791-b9df-bcd217f4368c","Type":"ContainerStarted","Data":"aa1a985ee96267d742d8dc091e2d0c3c3cb08ec55b1f7d1dd44cfbe8f42ff242"} Oct 31 00:49:36.785963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:36.785929 4969 generic.go:356] "Generic (PLEG): container finished" podID="13c148b9-60f6-4df0-a473-dd731078121d" containerID="d2fe078e2bd57fa53f8a3dbba55b788fc08fd4d893d8dcd7322cdc45e1af12d0" exitCode=0 Oct 31 00:49:36.786080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:36.785978 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" event={"ID":"13c148b9-60f6-4df0-a473-dd731078121d","Type":"ContainerDied","Data":"d2fe078e2bd57fa53f8a3dbba55b788fc08fd4d893d8dcd7322cdc45e1af12d0"} Oct 31 00:49:36.786080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:36.786019 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" event={"ID":"13c148b9-60f6-4df0-a473-dd731078121d","Type":"ContainerStarted","Data":"bce749cb48f22b52d049da57c53253d5b1ee63e970c7301b9ee2c0a710a9c036"} Oct 31 00:49:36.909140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:36.909075 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" podStartSLOduration=2.90905595 podStartE2EDuration="2.90905595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:36.907291047 +0000 UTC m=+2809.382649472" watchObservedRunningTime="2026-10-31 00:49:36.90905595 +0000 UTC m=+2809.384414374" Oct 31 00:49:37.794712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:37.794675 4969 generic.go:356] "Generic (PLEG): container finished" podID="531a267e-a00f-4791-b9df-bcd217f4368c" containerID="9f5d1b10b92d33939b2eca44852c52d95a7961058d34aa42f96ca78acda557c5" exitCode=0 Oct 31 00:49:37.795171 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:37.794764 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" event={"ID":"531a267e-a00f-4791-b9df-bcd217f4368c","Type":"ContainerDied","Data":"9f5d1b10b92d33939b2eca44852c52d95a7961058d34aa42f96ca78acda557c5"} Oct 31 00:49:37.948861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:37.948821 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:38.082077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.082005 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-openshift-ca\") pod \"13c148b9-60f6-4df0-a473-dd731078121d\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " Oct 31 00:49:38.082077 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.082046 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-xks-ca\") pod \"13c148b9-60f6-4df0-a473-dd731078121d\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " Oct 31 00:49:38.082269 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.082189 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c148b9-60f6-4df0-a473-dd731078121d-kube-api-access-57n5b\" (UniqueName: \"kubernetes.io/projected/13c148b9-60f6-4df0-a473-dd731078121d-kube-api-access-57n5b\") pod \"13c148b9-60f6-4df0-a473-dd731078121d\" (UID: \"13c148b9-60f6-4df0-a473-dd731078121d\") " Oct 31 00:49:38.082430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.082404 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-openshift-ca" pod "13c148b9-60f6-4df0-a473-dd731078121d" (UID: "13c148b9-60f6-4df0-a473-dd731078121d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:38.082641 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.082624 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:38.084416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.084380 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-xks-ca" pod "13c148b9-60f6-4df0-a473-dd731078121d" (UID: "13c148b9-60f6-4df0-a473-dd731078121d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:38.084518 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.084446 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c148b9-60f6-4df0-a473-dd731078121d-kube-api-access-57n5b" pod "13c148b9-60f6-4df0-a473-dd731078121d" (UID: "13c148b9-60f6-4df0-a473-dd731078121d"). InnerVolumeSpecName "kube-api-access-57n5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:38.184084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.184052 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57n5b\" (UniqueName: \"kubernetes.io/projected/13c148b9-60f6-4df0-a473-dd731078121d-kube-api-access-57n5b\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:38.184084 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.184082 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c148b9-60f6-4df0-a473-dd731078121d-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:38.803033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.803000 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" Oct 31 00:49:38.803404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.803032 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj" event={"ID":"13c148b9-60f6-4df0-a473-dd731078121d","Type":"ContainerDied","Data":"bce749cb48f22b52d049da57c53253d5b1ee63e970c7301b9ee2c0a710a9c036"} Oct 31 00:49:38.803404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.803080 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bce749cb48f22b52d049da57c53253d5b1ee63e970c7301b9ee2c0a710a9c036" Oct 31 00:49:38.940080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:38.940055 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:39.092148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.092056 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-openshift-ca\") pod \"531a267e-a00f-4791-b9df-bcd217f4368c\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " Oct 31 00:49:39.092148 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.092107 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531a267e-a00f-4791-b9df-bcd217f4368c-kube-api-access-c28dk\" (UniqueName: \"kubernetes.io/projected/531a267e-a00f-4791-b9df-bcd217f4368c-kube-api-access-c28dk\") pod \"531a267e-a00f-4791-b9df-bcd217f4368c\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " Oct 31 00:49:39.092377 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.092222 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-xks-ca\") pod \"531a267e-a00f-4791-b9df-bcd217f4368c\" (UID: \"531a267e-a00f-4791-b9df-bcd217f4368c\") " Oct 31 00:49:39.092477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.092435 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-openshift-ca" pod "531a267e-a00f-4791-b9df-bcd217f4368c" (UID: "531a267e-a00f-4791-b9df-bcd217f4368c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:39.092843 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.092823 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:39.094902 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.094878 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531a267e-a00f-4791-b9df-bcd217f4368c-kube-api-access-c28dk" pod "531a267e-a00f-4791-b9df-bcd217f4368c" (UID: "531a267e-a00f-4791-b9df-bcd217f4368c"). InnerVolumeSpecName "kube-api-access-c28dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:39.098939 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.098918 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-xks-ca" pod "531a267e-a00f-4791-b9df-bcd217f4368c" (UID: "531a267e-a00f-4791-b9df-bcd217f4368c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:39.193668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.193635 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531a267e-a00f-4791-b9df-bcd217f4368c-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:39.193668 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.193664 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c28dk\" (UniqueName: \"kubernetes.io/projected/531a267e-a00f-4791-b9df-bcd217f4368c-kube-api-access-c28dk\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:39.812250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.812219 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" Oct 31 00:49:39.812250 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.812240 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2" event={"ID":"531a267e-a00f-4791-b9df-bcd217f4368c","Type":"ContainerDied","Data":"aa1a985ee96267d742d8dc091e2d0c3c3cb08ec55b1f7d1dd44cfbe8f42ff242"} Oct 31 00:49:39.812736 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:39.812290 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa1a985ee96267d742d8dc091e2d0c3c3cb08ec55b1f7d1dd44cfbe8f42ff242" Oct 31 00:49:41.799819 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.799776 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d"] Oct 31 00:49:41.800536 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.800513 4969 cpu_manager.go:414] "containerMap: removing container" podUID="531a267e-a00f-4791-b9df-bcd217f4368c" containerName="revoke-keys" Oct 31 00:49:41.800536 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.800535 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531a267e-a00f-4791-b9df-bcd217f4368c" containerName="revoke-keys" Oct 31 00:49:41.800763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.800639 4969 cpu_manager.go:414] "containerMap: removing container" podUID="13c148b9-60f6-4df0-a473-dd731078121d" containerName="revoke-keys" Oct 31 00:49:41.800763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.800649 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c148b9-60f6-4df0-a473-dd731078121d" containerName="revoke-keys" Oct 31 00:49:41.800920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.800795 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c148b9-60f6-4df0-a473-dd731078121d" containerName="revoke-keys" Oct 31 00:49:41.800920 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.800811 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="531a267e-a00f-4791-b9df-bcd217f4368c" containerName="revoke-keys" Oct 31 00:49:41.804337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.804316 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:41.909558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.909528 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d"] Oct 31 00:49:41.924721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.924677 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:41.924921 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.924732 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742f0e21-1b98-4190-a352-262d1733434a-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:41.924921 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.924844 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:41.924921 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.924898 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742f0e21-1b98-4190-a352-262d1733434a-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:41.925040 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.924968 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:41.925079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.925034 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:41.925141 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.925118 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbrs\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-kube-api-access-7xbrs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:41.925182 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.925155 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:41.925230 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:41.925183 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.026704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.026668 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.026906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.026738 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbrs\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-kube-api-access-7xbrs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.026906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.026769 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.026906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.026806 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.026906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.026876 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.027129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.026911 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742f0e21-1b98-4190-a352-262d1733434a-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.027129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.026961 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.027129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.027013 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742f0e21-1b98-4190-a352-262d1733434a-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.027129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.027041 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.027323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.027134 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.027323 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.027189 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.027448 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.027320 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.027510 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.027479 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.027955 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.027935 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742f0e21-1b98-4190-a352-262d1733434a-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.029370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.029353 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.030020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.029996 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742f0e21-1b98-4190-a352-262d1733434a-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.087416 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.087353 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.087912 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.087894 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbrs\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-kube-api-access-7xbrs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.117378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.117351 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:42.551914 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.551884 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d"] Oct 31 00:49:42.554392 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:49:42.554351 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742f0e21_1b98_4190_a352_262d1733434a.slice/crio-a2472bea8ea8bc4b4a43929542deefe7e70dadf5fa7e7684569c2df5705338f2 WatchSource:0}: Error finding container a2472bea8ea8bc4b4a43929542deefe7e70dadf5fa7e7684569c2df5705338f2: Status 404 returned error can't find the container with id a2472bea8ea8bc4b4a43929542deefe7e70dadf5fa7e7684569c2df5705338f2 Oct 31 00:49:42.556869 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.556830 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:42.556947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.556902 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:42.556947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.556930 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:42.840331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.840233 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" event={"ID":"742f0e21-1b98-4190-a352-262d1733434a","Type":"ContainerStarted","Data":"3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52"} Oct 31 00:49:42.840331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.840297 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" event={"ID":"742f0e21-1b98-4190-a352-262d1733434a","Type":"ContainerStarted","Data":"a2472bea8ea8bc4b4a43929542deefe7e70dadf5fa7e7684569c2df5705338f2"} Oct 31 00:49:42.963942 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:42.963890 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podStartSLOduration=1.963875569 podStartE2EDuration="1.963875569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:42.959223703 +0000 UTC m=+2815.434582128" watchObservedRunningTime="2026-10-31 00:49:42.963875569 +0000 UTC m=+2815.439233992" Oct 31 00:49:43.118165 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.118128 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:43.119248 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.119223 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 00:49:43.119383 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.119284 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 00:49:43.758296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.758269 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x"] Oct 31 00:49:43.762274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.762259 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.847844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.847811 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cfc055-openshift-default-dockercfg-ft79f\"" Oct 31 00:49:43.854787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.853971 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qts\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-kube-api-access-n7qts\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.854787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.854020 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-podinfo\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.854787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.854071 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-socket\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.854787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.854100 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.854787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.854134 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-certs\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.854787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.854195 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-data\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.854787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.854378 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-credential-socket\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.854787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.854488 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-token\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.854787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.854517 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-envoy\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.870863 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.870834 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x"] Oct 31 00:49:43.955769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.955729 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qts\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-kube-api-access-n7qts\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.955769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.955767 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-podinfo\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.955810 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-socket\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.955832 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.955856 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-certs\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.955922 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-data\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.955985 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-credential-socket\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.956073 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-token\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.956103 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-envoy\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.956271 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-socket\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.956359 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-certs\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956559 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.956535 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-credential-socket\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.956552 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-data\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.956962 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.956946 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.958680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.958659 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-envoy\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:43.958908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:43.958889 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-podinfo\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:44.019111 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.018989 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-token\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:44.019852 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.019824 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qts\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-kube-api-access-n7qts\") pod \"e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:44.104027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.103993 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:44.118547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.118514 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 00:49:44.118686 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.118579 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 00:49:44.539894 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.539865 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x"] Oct 31 00:49:44.539894 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:49:44.539880 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c3fc30_57ad_4e6f_9254_d8ee253d8d44.slice/crio-d3aeb8ea600df95b5fd99636aa7176d85473edff0deff2359c69240aadeb3dad WatchSource:0}: Error finding container d3aeb8ea600df95b5fd99636aa7176d85473edff0deff2359c69240aadeb3dad: Status 404 returned error can't find the container with id d3aeb8ea600df95b5fd99636aa7176d85473edff0deff2359c69240aadeb3dad Oct 31 00:49:44.547913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.547864 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:44.548020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.547963 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:44.548020 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.548005 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:44.859795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.859690 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" event={"ID":"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44","Type":"ContainerStarted","Data":"52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447"} Oct 31 00:49:44.859795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:44.859786 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" event={"ID":"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44","Type":"ContainerStarted","Data":"d3aeb8ea600df95b5fd99636aa7176d85473edff0deff2359c69240aadeb3dad"} Oct 31 00:49:45.107942 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:45.107690 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:45.116887 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:45.116859 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:45.118268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:45.118241 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 00:49:45.118608 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:45.118292 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 00:49:45.241023 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:45.240968 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" podStartSLOduration=2.240953877 podStartE2EDuration="2.240953877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:44.98479226 +0000 UTC m=+2817.460150695" watchObservedRunningTime="2026-10-31 00:49:45.240953877 +0000 UTC m=+2817.716312300" Oct 31 00:49:45.873659 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:45.873621 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:45.901628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:45.901396 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:49:46.021036 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.020999 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb"] Oct 31 00:49:46.025029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.025003 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.089957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.084840 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:46.089957 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.085000 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:49:46.108055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.108008 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8f2378-22zcb\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.108243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.108163 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8f2378-22zcb\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.108384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.108354 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmlc\" (UniqueName: \"kubernetes.io/projected/75d82be4-6d76-450a-a186-6f3bd7c52ef4-kube-api-access-wlmlc\") pod \"maas-api-revoke-sub-3d8f2378-22zcb\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.111538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.111394 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb"] Oct 31 00:49:46.118446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.118416 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 00:49:46.118560 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.118482 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 00:49:46.209889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.209793 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmlc\" (UniqueName: \"kubernetes.io/projected/75d82be4-6d76-450a-a186-6f3bd7c52ef4-kube-api-access-wlmlc\") pod \"maas-api-revoke-sub-3d8f2378-22zcb\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.209889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.209886 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8f2378-22zcb\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.210116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.209949 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8f2378-22zcb\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.210570 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.210545 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8f2378-22zcb\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.212540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.212516 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8f2378-22zcb\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.270908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.270882 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmlc\" (UniqueName: \"kubernetes.io/projected/75d82be4-6d76-450a-a186-6f3bd7c52ef4-kube-api-access-wlmlc\") pod \"maas-api-revoke-sub-3d8f2378-22zcb\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.340069 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.340028 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:46.766073 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:49:46.766041 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d82be4_6d76_450a_a186_6f3bd7c52ef4.slice/crio-362ae772be2cb4395c7f26c7d7d1da6373ae39eddc3989f0b82412fb717519c7 WatchSource:0}: Error finding container 362ae772be2cb4395c7f26c7d7d1da6373ae39eddc3989f0b82412fb717519c7: Status 404 returned error can't find the container with id 362ae772be2cb4395c7f26c7d7d1da6373ae39eddc3989f0b82412fb717519c7 Oct 31 00:49:46.766433 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.766404 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb"] Oct 31 00:49:46.902629 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:46.900529 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" event={"ID":"75d82be4-6d76-450a-a186-6f3bd7c52ef4","Type":"ContainerStarted","Data":"362ae772be2cb4395c7f26c7d7d1da6373ae39eddc3989f0b82412fb717519c7"} Oct 31 00:49:47.117945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.117913 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 00:49:47.118134 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.117967 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 00:49:47.570680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.570639 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g"] Oct 31 00:49:47.574643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.574620 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.626044 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.626015 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-socket\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.626211 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.626134 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-token\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.626275 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.626217 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-certs\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.626275 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.626235 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-envoy\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.626376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.626332 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-podinfo\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.626450 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.626426 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-credential-socket\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.626508 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.626468 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgbg\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-kube-api-access-crgbg\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.626548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.626509 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istiod-ca-cert\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.626660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.626636 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-data\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.628493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.628468 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f13553-openshift-default-dockercfg-xwg5l\"" Oct 31 00:49:47.705692 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.705660 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g"] Oct 31 00:49:47.728276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.727837 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-certs\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.728276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.727886 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-envoy\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.728276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.727917 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-podinfo\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.728276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.727955 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-credential-socket\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.728276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.727980 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgbg\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-kube-api-access-crgbg\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.728276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.728013 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istiod-ca-cert\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.728276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.728075 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-data\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.728276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.728105 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-socket\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.728276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.728209 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-token\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.729159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.729134 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-certs\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.740098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.734689 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-data\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.740098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.735490 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-credential-socket\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.740098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.735863 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-socket\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.740098 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.736481 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istiod-ca-cert\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.742761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.742735 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-envoy\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.759399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.759234 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-podinfo\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.811222 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.811196 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgbg\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-kube-api-access-crgbg\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.811410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.811274 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-token\") pod \"e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:47.914374 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.914338 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" event={"ID":"75d82be4-6d76-450a-a186-6f3bd7c52ef4","Type":"ContainerStarted","Data":"67210ec5337b6f3b4ab09364e69e1e25eb14d537bfb3bf562934f2ef2e6ae85b"} Oct 31 00:49:47.919778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:47.919746 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:48.043734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.043680 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" podStartSLOduration=3.043664815 podStartE2EDuration="3.043664815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:48.038079588 +0000 UTC m=+2820.513438011" watchObservedRunningTime="2026-10-31 00:49:48.043664815 +0000 UTC m=+2820.519023238" Oct 31 00:49:48.118337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.118302 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 00:49:48.118527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.118362 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 00:49:48.403340 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.403309 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g"] Oct 31 00:49:48.405693 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:49:48.405661 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4767fc84_2b65_4ec8_87cd_e7e2f1c2f4e0.slice/crio-2e8585d37a9b04143861799674e3e90f3b5d6ba5ec1246621ea3e48c38dafda9 WatchSource:0}: Error finding container 2e8585d37a9b04143861799674e3e90f3b5d6ba5ec1246621ea3e48c38dafda9: Status 404 returned error can't find the container with id 2e8585d37a9b04143861799674e3e90f3b5d6ba5ec1246621ea3e48c38dafda9 Oct 31 00:49:48.408662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.408629 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:48.408769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.408716 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:48.408769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.408748 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:48.932811 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.932771 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" event={"ID":"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0","Type":"ContainerStarted","Data":"efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845"} Oct 31 00:49:48.933321 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.932869 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" event={"ID":"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0","Type":"ContainerStarted","Data":"2e8585d37a9b04143861799674e3e90f3b5d6ba5ec1246621ea3e48c38dafda9"} Oct 31 00:49:48.934905 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.934875 4969 generic.go:356] "Generic (PLEG): container finished" podID="75d82be4-6d76-450a-a186-6f3bd7c52ef4" containerID="67210ec5337b6f3b4ab09364e69e1e25eb14d537bfb3bf562934f2ef2e6ae85b" exitCode=0 Oct 31 00:49:48.935045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:48.934931 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" event={"ID":"75d82be4-6d76-450a-a186-6f3bd7c52ef4","Type":"ContainerDied","Data":"67210ec5337b6f3b4ab09364e69e1e25eb14d537bfb3bf562934f2ef2e6ae85b"} Oct 31 00:49:49.058765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:49.058714 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" podStartSLOduration=2.058700007 podStartE2EDuration="2.058700007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:49.054317669 +0000 UTC m=+2821.529676092" watchObservedRunningTime="2026-10-31 00:49:49.058700007 +0000 UTC m=+2821.534058427" Oct 31 00:49:49.117779 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:49.117726 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 31 00:49:49.118042 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:49.117790 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 31 00:49:49.920463 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:49.920410 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:49.927136 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:49.927104 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:49.941964 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:49.941762 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:49.942524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:49.942504 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:49:50.101889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.101865 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:50.122873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.122848 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:50.123166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.123143 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:50.123854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.123840 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:49:50.160168 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.160140 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-openshift-ca\") pod \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " Oct 31 00:49:50.160283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.160200 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-xks-ca\") pod \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " Oct 31 00:49:50.160283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.160257 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d82be4-6d76-450a-a186-6f3bd7c52ef4-kube-api-access-wlmlc\" (UniqueName: \"kubernetes.io/projected/75d82be4-6d76-450a-a186-6f3bd7c52ef4-kube-api-access-wlmlc\") pod \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\" (UID: \"75d82be4-6d76-450a-a186-6f3bd7c52ef4\") " Oct 31 00:49:50.160512 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.160487 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-openshift-ca" pod "75d82be4-6d76-450a-a186-6f3bd7c52ef4" (UID: "75d82be4-6d76-450a-a186-6f3bd7c52ef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:50.160709 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.160685 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:50.162545 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.162522 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-xks-ca" pod "75d82be4-6d76-450a-a186-6f3bd7c52ef4" (UID: "75d82be4-6d76-450a-a186-6f3bd7c52ef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:50.162770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.162752 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d82be4-6d76-450a-a186-6f3bd7c52ef4-kube-api-access-wlmlc" pod "75d82be4-6d76-450a-a186-6f3bd7c52ef4" (UID: "75d82be4-6d76-450a-a186-6f3bd7c52ef4"). InnerVolumeSpecName "kube-api-access-wlmlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:50.261931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.261903 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d82be4-6d76-450a-a186-6f3bd7c52ef4-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:50.261931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.261929 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlmlc\" (UniqueName: \"kubernetes.io/projected/75d82be4-6d76-450a-a186-6f3bd7c52ef4-kube-api-access-wlmlc\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:49:50.951881 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.951853 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" Oct 31 00:49:50.952389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.951852 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb" event={"ID":"75d82be4-6d76-450a-a186-6f3bd7c52ef4","Type":"ContainerDied","Data":"362ae772be2cb4395c7f26c7d7d1da6373ae39eddc3989f0b82412fb717519c7"} Oct 31 00:49:50.952389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:50.951962 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="362ae772be2cb4395c7f26c7d7d1da6373ae39eddc3989f0b82412fb717519c7" Oct 31 00:49:52.896290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:52.896251 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp"] Oct 31 00:49:52.897054 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:52.897031 4969 cpu_manager.go:414] "containerMap: removing container" podUID="75d82be4-6d76-450a-a186-6f3bd7c52ef4" containerName="revoke-keys" Oct 31 00:49:52.897159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:52.897055 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d82be4-6d76-450a-a186-6f3bd7c52ef4" containerName="revoke-keys" Oct 31 00:49:52.897227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:52.897211 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d82be4-6d76-450a-a186-6f3bd7c52ef4" containerName="revoke-keys" Oct 31 00:49:52.900837 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:52.900815 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:49:52.954874 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:52.954848 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-cfc055\"" Oct 31 00:49:52.960004 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:52.959976 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp"] Oct 31 00:49:52.992854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:52.992817 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp\" (UID: \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:49:52.993066 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:52.993047 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-kube-api-access-bwgkm\") pod \"payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp\" (UID: \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:49:53.094704 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:53.094673 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-kube-api-access-bwgkm\") pod \"payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp\" (UID: \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:49:53.094870 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:53.094738 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp\" (UID: \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:49:53.095458 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:53.095436 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp\" (UID: \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:49:53.180782 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:53.180704 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-kube-api-access-bwgkm\") pod \"payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp\" (UID: \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:49:53.214873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:53.214834 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:49:53.599834 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:53.599801 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp"] Oct 31 00:49:53.600488 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:49:53.600463 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092b2bde_bf8c_4cf0_b41d_1bd6c323e6fc.slice/crio-3f87a967d754b9a386e954220ed5f58f8e0ae6041f8deed1962f63e90e56a22f WatchSource:0}: Error finding container 3f87a967d754b9a386e954220ed5f58f8e0ae6041f8deed1962f63e90e56a22f: Status 404 returned error can't find the container with id 3f87a967d754b9a386e954220ed5f58f8e0ae6041f8deed1962f63e90e56a22f Oct 31 00:49:53.974607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:53.974558 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" event={"ID":"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc","Type":"ContainerStarted","Data":"29fd64bbb9237c5438cbc0349f34345b8cff357121983a0e077c214e8cc867c2"} Oct 31 00:49:53.975078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:53.974645 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" event={"ID":"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc","Type":"ContainerStarted","Data":"3f87a967d754b9a386e954220ed5f58f8e0ae6041f8deed1962f63e90e56a22f"} Oct 31 00:49:53.975078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:53.974674 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:49:54.094144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:49:54.094082 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" podStartSLOduration=2.094065344 podStartE2EDuration="2.094065344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:54.093181352 +0000 UTC m=+2826.568539825" watchObservedRunningTime="2026-10-31 00:49:54.094065344 +0000 UTC m=+2826.569423770" Oct 31 00:50:00.381780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.381537 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm"] Oct 31 00:50:00.398893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.398862 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.458925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.458869 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-16b984-openshift-default-dockercfg-w2jsh\"" Oct 31 00:50:00.526622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.523264 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7697441d-50ec-428c-90ef-1286c6498dc6-istio-podinfo\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.534770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.534725 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-data\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.557754 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.557709 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w54v6\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-kube-api-access-w54v6\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.558076 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.558050 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-credential-socket\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.558245 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.558224 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7697441d-50ec-428c-90ef-1286c6498dc6-istiod-ca-cert\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.558412 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.558389 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-envoy\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.561617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.558608 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-istio-token\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.561858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.561834 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-socket\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.562038 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.562016 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-certs\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.664745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.664654 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm"] Oct 31 00:50:00.718617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.689183 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7697441d-50ec-428c-90ef-1286c6498dc6-istio-podinfo\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.740048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.739995 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-data\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.767620 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.743692 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w54v6\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-kube-api-access-w54v6\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.767943 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.767912 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-credential-socket\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.768128 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.768109 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7697441d-50ec-428c-90ef-1286c6498dc6-istiod-ca-cert\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.768276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.768259 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-envoy\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.768471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.768451 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-istio-token\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.768685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.768665 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-socket\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.768866 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.768837 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-certs\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.775145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.769341 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-certs\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.775145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.770993 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-credential-socket\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.775145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.745054 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-data\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.775145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.766272 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7697441d-50ec-428c-90ef-1286c6498dc6-istio-podinfo\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.775145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.771961 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-socket\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.775145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.772325 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7697441d-50ec-428c-90ef-1286c6498dc6-istiod-ca-cert\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.775145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.774889 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-envoy\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.917652 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.902714 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w54v6\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-kube-api-access-w54v6\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:00.941833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:00.941797 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-istio-token\") pod \"e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:01.060078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:01.060035 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:01.667285 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:01.667258 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm"] Oct 31 00:50:01.668156 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:50:01.668135 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7697441d_50ec_428c_90ef_1286c6498dc6.slice/crio-9746d13f8ff3084e77fbfa26f8d76536cfc7002f51f14478e00cb7152be74d6c WatchSource:0}: Error finding container 9746d13f8ff3084e77fbfa26f8d76536cfc7002f51f14478e00cb7152be74d6c: Status 404 returned error can't find the container with id 9746d13f8ff3084e77fbfa26f8d76536cfc7002f51f14478e00cb7152be74d6c Oct 31 00:50:01.670660 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:01.670624 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:01.670756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:01.670687 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:01.670756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:01.670716 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:02.071742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.071659 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7"] Oct 31 00:50:02.076915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.076881 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" event={"ID":"7697441d-50ec-428c-90ef-1286c6498dc6","Type":"ContainerStarted","Data":"98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6"} Oct 31 00:50:02.077065 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.076929 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" event={"ID":"7697441d-50ec-428c-90ef-1286c6498dc6","Type":"ContainerStarted","Data":"9746d13f8ff3084e77fbfa26f8d76536cfc7002f51f14478e00cb7152be74d6c"} Oct 31 00:50:02.077065 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.077005 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.129384 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.129345 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:02.129580 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.129448 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:50:02.131527 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.131499 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7"] Oct 31 00:50:02.197372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.197314 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" podStartSLOduration=2.197299267 podStartE2EDuration="2.197299267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:02.194243549 +0000 UTC m=+2834.669601976" watchObservedRunningTime="2026-10-31 00:50:02.197299267 +0000 UTC m=+2834.672657743" Oct 31 00:50:02.228037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.228003 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86289c90-qvcf7\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.228268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.228244 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcvz\" (UniqueName: \"kubernetes.io/projected/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-kube-api-access-vfcvz\") pod \"maas-api-revoke-sub-86289c90-qvcf7\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.228329 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.228310 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86289c90-qvcf7\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.329484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.329391 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcvz\" (UniqueName: \"kubernetes.io/projected/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-kube-api-access-vfcvz\") pod \"maas-api-revoke-sub-86289c90-qvcf7\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.329484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.329442 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86289c90-qvcf7\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.329750 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.329559 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86289c90-qvcf7\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.330222 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.330199 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86289c90-qvcf7\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.332270 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.332241 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86289c90-qvcf7\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.390358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.390317 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcvz\" (UniqueName: \"kubernetes.io/projected/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-kube-api-access-vfcvz\") pod \"maas-api-revoke-sub-86289c90-qvcf7\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.394185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.394163 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:02.728378 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:50:02.728342 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:50:02.776208 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:50:02.776176 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4e4412_02a1_4942_9ee7_a77fd6a9133f.slice/crio-d326383a0511bb0206fcde928c7203321176f1cfc79a9e39401603f3a9ee5eea WatchSource:0}: Error finding container d326383a0511bb0206fcde928c7203321176f1cfc79a9e39401603f3a9ee5eea: Status 404 returned error can't find the container with id d326383a0511bb0206fcde928c7203321176f1cfc79a9e39401603f3a9ee5eea Oct 31 00:50:02.777057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:02.777022 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7"] Oct 31 00:50:03.060698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:03.060664 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:03.066812 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:03.066788 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:03.086542 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:03.086494 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" event={"ID":"2e4e4412-02a1-4942-9ee7-a77fd6a9133f","Type":"ContainerStarted","Data":"d326383a0511bb0206fcde928c7203321176f1cfc79a9e39401603f3a9ee5eea"} Oct 31 00:50:03.086734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:03.086577 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:03.087671 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:03.087651 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:50:04.094875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:04.094791 4969 generic.go:356] "Generic (PLEG): container finished" podID="2e4e4412-02a1-4942-9ee7-a77fd6a9133f" containerID="83ade6325d45670cc6bd2ba9db969fa30275288600c5954fa0ab02610ef06cb9" exitCode=0 Oct 31 00:50:04.095331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:04.094879 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" event={"ID":"2e4e4412-02a1-4942-9ee7-a77fd6a9133f","Type":"ContainerDied","Data":"83ade6325d45670cc6bd2ba9db969fa30275288600c5954fa0ab02610ef06cb9"} Oct 31 00:50:04.984956 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:04.984929 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 00:50:05.279675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.279651 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:05.466310 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.466271 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-openshift-ca\") pod \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " Oct 31 00:50:05.466515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.466318 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-xks-ca\") pod \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " Oct 31 00:50:05.466515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.466385 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-kube-api-access-vfcvz\" (UniqueName: \"kubernetes.io/projected/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-kube-api-access-vfcvz\") pod \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\" (UID: \"2e4e4412-02a1-4942-9ee7-a77fd6a9133f\") " Oct 31 00:50:05.466705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.466675 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-openshift-ca" pod "2e4e4412-02a1-4942-9ee7-a77fd6a9133f" (UID: "2e4e4412-02a1-4942-9ee7-a77fd6a9133f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:05.466833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.466815 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:05.468747 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.468726 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-xks-ca" pod "2e4e4412-02a1-4942-9ee7-a77fd6a9133f" (UID: "2e4e4412-02a1-4942-9ee7-a77fd6a9133f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:05.468832 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.468767 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-kube-api-access-vfcvz" pod "2e4e4412-02a1-4942-9ee7-a77fd6a9133f" (UID: "2e4e4412-02a1-4942-9ee7-a77fd6a9133f"). InnerVolumeSpecName "kube-api-access-vfcvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:05.567892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.567813 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfcvz\" (UniqueName: \"kubernetes.io/projected/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-kube-api-access-vfcvz\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:05.567892 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:05.567839 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e4e4412-02a1-4942-9ee7-a77fd6a9133f-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.111362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:06.111320 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" event={"ID":"2e4e4412-02a1-4942-9ee7-a77fd6a9133f","Type":"ContainerDied","Data":"d326383a0511bb0206fcde928c7203321176f1cfc79a9e39401603f3a9ee5eea"} Oct 31 00:50:06.111362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:06.111348 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7" Oct 31 00:50:06.111654 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:06.111387 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d326383a0511bb0206fcde928c7203321176f1cfc79a9e39401603f3a9ee5eea" Oct 31 00:50:13.677086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.677052 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4"] Oct 31 00:50:13.679484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.677628 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2e4e4412-02a1-4942-9ee7-a77fd6a9133f" containerName="revoke-keys" Oct 31 00:50:13.679484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.677640 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4e4412-02a1-4942-9ee7-a77fd6a9133f" containerName="revoke-keys" Oct 31 00:50:13.679484 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.677745 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4e4412-02a1-4942-9ee7-a77fd6a9133f" containerName="revoke-keys" Oct 31 00:50:13.681072 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.681056 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.739707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.739668 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4"] Oct 31 00:50:13.768789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.768756 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nk5\" (UniqueName: \"kubernetes.io/projected/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-kube-api-access-v7nk5\") pod \"maas-api-revoke-sub-91046e21-mmhr4\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.768971 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.768879 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91046e21-mmhr4\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.769070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.769056 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91046e21-mmhr4\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.783961 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.783934 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:13.784090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.783973 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 00:50:13.870767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.870736 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91046e21-mmhr4\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.870947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.870893 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91046e21-mmhr4\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.870947 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.870940 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nk5\" (UniqueName: \"kubernetes.io/projected/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-kube-api-access-v7nk5\") pod \"maas-api-revoke-sub-91046e21-mmhr4\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.871536 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.871467 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91046e21-mmhr4\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.873267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.873245 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91046e21-mmhr4\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.931909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.931840 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nk5\" (UniqueName: \"kubernetes.io/projected/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-kube-api-access-v7nk5\") pod \"maas-api-revoke-sub-91046e21-mmhr4\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:13.994808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:13.994781 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:14.378738 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:50:14.378707 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2e0dc7_3bca_4f79_9e03_d638f29e605e.slice/crio-85d44bbb6a57052d65b9a53f1c6136e748c4305a9949c9b6e548d56cc09f9b6b WatchSource:0}: Error finding container 85d44bbb6a57052d65b9a53f1c6136e748c4305a9949c9b6e548d56cc09f9b6b: Status 404 returned error can't find the container with id 85d44bbb6a57052d65b9a53f1c6136e748c4305a9949c9b6e548d56cc09f9b6b Oct 31 00:50:14.378891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:14.378865 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4"] Oct 31 00:50:15.177374 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:15.177336 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" event={"ID":"2e2e0dc7-3bca-4f79-9e03-d638f29e605e","Type":"ContainerStarted","Data":"8b69cf6817f27b5eaf85102c9ffbc8a216bad592459d107df53f64ee59cfad6c"} Oct 31 00:50:15.177830 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:15.177403 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" event={"ID":"2e2e0dc7-3bca-4f79-9e03-d638f29e605e","Type":"ContainerStarted","Data":"85d44bbb6a57052d65b9a53f1c6136e748c4305a9949c9b6e548d56cc09f9b6b"} Oct 31 00:50:15.295658 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:15.295584 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" podStartSLOduration=2.295567351 podStartE2EDuration="2.295567351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:15.292001412 +0000 UTC m=+2847.767359835" watchObservedRunningTime="2026-10-31 00:50:15.295567351 +0000 UTC m=+2847.770925774" Oct 31 00:50:16.185646 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:16.185582 4969 generic.go:356] "Generic (PLEG): container finished" podID="2e2e0dc7-3bca-4f79-9e03-d638f29e605e" containerID="8b69cf6817f27b5eaf85102c9ffbc8a216bad592459d107df53f64ee59cfad6c" exitCode=0 Oct 31 00:50:16.186058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:16.185685 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" event={"ID":"2e2e0dc7-3bca-4f79-9e03-d638f29e605e","Type":"ContainerDied","Data":"8b69cf6817f27b5eaf85102c9ffbc8a216bad592459d107df53f64ee59cfad6c"} Oct 31 00:50:17.358455 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.358429 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:17.414129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.414099 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-kube-api-access-v7nk5\" (UniqueName: \"kubernetes.io/projected/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-kube-api-access-v7nk5\") pod \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " Oct 31 00:50:17.414293 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.414166 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-openshift-ca\") pod \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " Oct 31 00:50:17.414293 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.414257 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-xks-ca\") pod \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\" (UID: \"2e2e0dc7-3bca-4f79-9e03-d638f29e605e\") " Oct 31 00:50:17.414554 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.414528 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-openshift-ca" pod "2e2e0dc7-3bca-4f79-9e03-d638f29e605e" (UID: "2e2e0dc7-3bca-4f79-9e03-d638f29e605e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:17.414827 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.414801 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:17.416475 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.416453 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-xks-ca" pod "2e2e0dc7-3bca-4f79-9e03-d638f29e605e" (UID: "2e2e0dc7-3bca-4f79-9e03-d638f29e605e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:17.416618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.416572 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-kube-api-access-v7nk5" pod "2e2e0dc7-3bca-4f79-9e03-d638f29e605e" (UID: "2e2e0dc7-3bca-4f79-9e03-d638f29e605e"). InnerVolumeSpecName "kube-api-access-v7nk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:17.515906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.515826 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nk5\" (UniqueName: \"kubernetes.io/projected/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-kube-api-access-v7nk5\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:17.515906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:17.515859 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2e0dc7-3bca-4f79-9e03-d638f29e605e-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:50:18.200372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:18.200328 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" event={"ID":"2e2e0dc7-3bca-4f79-9e03-d638f29e605e","Type":"ContainerDied","Data":"85d44bbb6a57052d65b9a53f1c6136e748c4305a9949c9b6e548d56cc09f9b6b"} Oct 31 00:50:18.200372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:18.200373 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4" Oct 31 00:50:18.200583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:18.200387 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d44bbb6a57052d65b9a53f1c6136e748c4305a9949c9b6e548d56cc09f9b6b" Oct 31 00:50:20.200833 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.200793 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6"] Oct 31 00:50:20.201535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.201512 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2e2e0dc7-3bca-4f79-9e03-d638f29e605e" containerName="revoke-keys" Oct 31 00:50:20.201675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.201536 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2e0dc7-3bca-4f79-9e03-d638f29e605e" containerName="revoke-keys" Oct 31 00:50:20.201776 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.201761 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2e0dc7-3bca-4f79-9e03-d638f29e605e" containerName="revoke-keys" Oct 31 00:50:20.205413 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.205392 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.257904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.257878 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-cfc055-serving-cert\"" Oct 31 00:50:20.307666 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.307637 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-442zl\"" Oct 31 00:50:20.307885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.307663 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:50:20.311823 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.311795 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6"] Oct 31 00:50:20.349913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.349879 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.350070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.349953 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntg9r\" (UniqueName: \"kubernetes.io/projected/08677af3-2a63-4aee-8507-95a1eab3cfcf-kube-api-access-ntg9r\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.350070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.350047 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.350152 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.350096 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08677af3-2a63-4aee-8507-95a1eab3cfcf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.451405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.451306 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.451405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.451369 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntg9r\" (UniqueName: \"kubernetes.io/projected/08677af3-2a63-4aee-8507-95a1eab3cfcf-kube-api-access-ntg9r\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.451405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.451391 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.452331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.451432 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08677af3-2a63-4aee-8507-95a1eab3cfcf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.452331 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:50:20.451463 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-serving-cert: secret "maas-api-e2e-shared-a-w0-cfc055-serving-cert" not found Oct 31 00:50:20.452331 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:50:20.451553 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls podName:08677af3-2a63-4aee-8507-95a1eab3cfcf nodeName:}" failed. No retries permitted until 2026-10-31 00:50:20.951531282 +0000 UTC m=+2853.426889685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls") pod "maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" (UID: "08677af3-2a63-4aee-8507-95a1eab3cfcf") : secret "maas-api-e2e-shared-a-w0-cfc055-serving-cert" not found Oct 31 00:50:20.452331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.452144 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08677af3-2a63-4aee-8507-95a1eab3cfcf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.464365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.464331 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.513789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.513754 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntg9r\" (UniqueName: \"kubernetes.io/projected/08677af3-2a63-4aee-8507-95a1eab3cfcf-kube-api-access-ntg9r\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.958778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.958739 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:20.961333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:20.961307 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:21.048662 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:21.048619 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf"] Oct 31 00:50:21.052105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:21.052084 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a05595-hgdrf"] Oct 31 00:50:21.122507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:21.122470 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:21.552296 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:21.552265 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6"] Oct 31 00:50:21.553135 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:50:21.553106 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08677af3_2a63_4aee_8507_95a1eab3cfcf.slice/crio-ad321a5b27e33a30306796558950d49dc454c089c97df1d4274bb0c672a51869 WatchSource:0}: Error finding container ad321a5b27e33a30306796558950d49dc454c089c97df1d4274bb0c672a51869: Status 404 returned error can't find the container with id ad321a5b27e33a30306796558950d49dc454c089c97df1d4274bb0c672a51869 Oct 31 00:50:21.554700 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:21.554682 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:50:21.735793 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:21.735761 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cb07f1-5589-4157-978b-f25e1ca09640" path="/var/lib/kubelet/pods/c3cb07f1-5589-4157-978b-f25e1ca09640/volumes" Oct 31 00:50:22.234492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:22.234457 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" event={"ID":"08677af3-2a63-4aee-8507-95a1eab3cfcf","Type":"ContainerStarted","Data":"ad321a5b27e33a30306796558950d49dc454c089c97df1d4274bb0c672a51869"} Oct 31 00:50:23.244653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:23.244569 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" event={"ID":"08677af3-2a63-4aee-8507-95a1eab3cfcf","Type":"ContainerStarted","Data":"f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9"} Oct 31 00:50:23.245124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:23.244675 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:23.363719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:23.363665 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" podStartSLOduration=2.5118666899999997 podStartE2EDuration="3.363651285s" totalImagesPullingTime="851.784595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:23.361354179 +0000 UTC m=+2855.836712601" watchObservedRunningTime="2026-10-31 00:50:23.363651285 +0000 UTC m=+2855.839009723" Oct 31 00:50:25.548743 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:25.548686 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:26.822750 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.822709 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn"] Oct 31 00:50:26.827896 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.827875 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.891773 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.891732 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-4743f8-openshift-default-dockercfg-gftsq\"" Oct 31 00:50:26.923285 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.923249 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr24n\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-kube-api-access-mr24n\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.923487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.923332 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-envoy\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.923487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.923362 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-podinfo\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.923685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.923545 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-socket\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.923685 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.923656 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istiod-ca-cert\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.923807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.923749 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-certs\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.923807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.923775 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-credential-socket\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.923918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.923817 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-token\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.923918 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.923874 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-data\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:26.934544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:26.934513 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn"] Oct 31 00:50:27.025035 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.024998 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-socket\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025079 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istiod-ca-cert\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025172 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-certs\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025200 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-credential-socket\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025238 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-token\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025290 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-data\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025350 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr24n\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-kube-api-access-mr24n\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025372 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-socket\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025383 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-envoy\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025460 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-podinfo\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025515 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-certs\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025606 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-credential-socket\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.025739 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025696 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-data\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.026007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.025836 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istiod-ca-cert\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.028002 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.027974 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-envoy\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.028238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.028215 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-podinfo\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.086392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.086320 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-token\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.086517 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.086499 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr24n\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-kube-api-access-mr24n\") pod \"e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.142884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.142852 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:27.576104 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.576072 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn"] Oct 31 00:50:27.576308 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:50:27.576286 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcd577f_e015_4cbc_a379_d8ba5fe22385.slice/crio-1f5fb8d8fca6fed68f719257dbebb33507e5fbc3d497c1cbc71f4e4b41c9c0f7 WatchSource:0}: Error finding container 1f5fb8d8fca6fed68f719257dbebb33507e5fbc3d497c1cbc71f4e4b41c9c0f7: Status 404 returned error can't find the container with id 1f5fb8d8fca6fed68f719257dbebb33507e5fbc3d497c1cbc71f4e4b41c9c0f7 Oct 31 00:50:27.578653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.578623 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:27.578725 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.578690 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:27.578725 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:27.578722 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:28.337653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:28.337616 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" event={"ID":"0dcd577f-e015-4cbc-a379-d8ba5fe22385","Type":"ContainerStarted","Data":"a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe"} Oct 31 00:50:28.338177 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:28.337701 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" event={"ID":"0dcd577f-e015-4cbc-a379-d8ba5fe22385","Type":"ContainerStarted","Data":"1f5fb8d8fca6fed68f719257dbebb33507e5fbc3d497c1cbc71f4e4b41c9c0f7"} Oct 31 00:50:28.459211 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:28.459163 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" podStartSLOduration=2.459150149 podStartE2EDuration="2.459150149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:28.456911575 +0000 UTC m=+2860.932270000" watchObservedRunningTime="2026-10-31 00:50:28.459150149 +0000 UTC m=+2860.934508572" Oct 31 00:50:29.143626 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:29.143575 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:29.148908 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:29.148881 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:29.281889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:29.281862 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 00:50:29.348150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:29.348120 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:29.349222 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:29.349198 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:50:32.728451 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:50:32.728417 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:50:38.054567 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.054528 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9"] Oct 31 00:50:38.059909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.059892 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.113513 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.113481 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-1a1181-openshift-default-dockercfg-7r8ff\"" Oct 31 00:50:38.167979 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.167947 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9"] Oct 31 00:50:38.197163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.197123 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-credential-socket\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.197523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.197491 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-data\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.197745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.197549 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-token\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.197745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.197580 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-podinfo\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.197745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.197706 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2sjn\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-kube-api-access-l2sjn\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.197745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.197735 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-certs\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.198078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.197777 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istiod-ca-cert\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.198078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.197834 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-socket\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.198078 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.197880 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-envoy\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.299581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.299549 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2sjn\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-kube-api-access-l2sjn\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.299804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.299584 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-certs\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.299804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.299666 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istiod-ca-cert\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.299804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.299715 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-socket\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.299804 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.299753 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-envoy\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.300048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.299831 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-credential-socket\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.300048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.299919 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-data\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.300048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.299951 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-token\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.300048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.299982 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-podinfo\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.300276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.300060 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-certs\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.300276 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.300117 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-socket\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.300409 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.300273 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-credential-socket\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.300504 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.300484 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-data\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.300568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.300530 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istiod-ca-cert\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.302186 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.302168 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-envoy\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.302569 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.302552 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-podinfo\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.361096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.361035 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2sjn\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-kube-api-access-l2sjn\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.365365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.365338 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-token\") pod \"e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.374328 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.374307 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:38.808707 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:50:38.808673 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c58d5b_e672_4e93_8bfe_4f7677c31125.slice/crio-84cc014cb1fb3468ad7f9d2427f00704f1572f23a22be91b4f6d1bca1fb93404 WatchSource:0}: Error finding container 84cc014cb1fb3468ad7f9d2427f00704f1572f23a22be91b4f6d1bca1fb93404: Status 404 returned error can't find the container with id 84cc014cb1fb3468ad7f9d2427f00704f1572f23a22be91b4f6d1bca1fb93404 Oct 31 00:50:38.809881 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.809858 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9"] Oct 31 00:50:38.810963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.810934 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:38.811019 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.810996 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:38.811055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:38.811025 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:39.416698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:39.416664 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" event={"ID":"e7c58d5b-e672-4e93-8bfe-4f7677c31125","Type":"ContainerStarted","Data":"e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9"} Oct 31 00:50:39.417191 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:39.416717 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" event={"ID":"e7c58d5b-e672-4e93-8bfe-4f7677c31125","Type":"ContainerStarted","Data":"84cc014cb1fb3468ad7f9d2427f00704f1572f23a22be91b4f6d1bca1fb93404"} Oct 31 00:50:39.543212 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:39.543157 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" podStartSLOduration=1.543138845 podStartE2EDuration="1.543138845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:39.536459309 +0000 UTC m=+2872.011817730" watchObservedRunningTime="2026-10-31 00:50:39.543138845 +0000 UTC m=+2872.018497272" Oct 31 00:50:40.374775 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:40.374733 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:40.381150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:40.381126 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:40.425829 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:40.425802 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:40.426910 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:40.426886 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:50:46.222675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.222438 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj"] Oct 31 00:50:46.237140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.237112 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:46.271471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.271442 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj"] Oct 31 00:50:46.321027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.320992 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1a1181-dockercfg-6fww2\"" Oct 31 00:50:46.321286 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.321002 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1a1181\"" Oct 31 00:50:46.322118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.322090 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41835b87-f8c5-49d1-8a70-1ce7bf105487-plugins-config-volume\") pod \"payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj\" (UID: \"41835b87-f8c5-49d1-8a70-1ce7bf105487\") " pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:46.322246 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.322132 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66r9s\" (UniqueName: \"kubernetes.io/projected/41835b87-f8c5-49d1-8a70-1ce7bf105487-kube-api-access-66r9s\") pod \"payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj\" (UID: \"41835b87-f8c5-49d1-8a70-1ce7bf105487\") " pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:46.423581 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.423543 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41835b87-f8c5-49d1-8a70-1ce7bf105487-plugins-config-volume\") pod \"payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj\" (UID: \"41835b87-f8c5-49d1-8a70-1ce7bf105487\") " pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:46.423758 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.423632 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66r9s\" (UniqueName: \"kubernetes.io/projected/41835b87-f8c5-49d1-8a70-1ce7bf105487-kube-api-access-66r9s\") pod \"payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj\" (UID: \"41835b87-f8c5-49d1-8a70-1ce7bf105487\") " pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:46.424374 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.424322 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41835b87-f8c5-49d1-8a70-1ce7bf105487-plugins-config-volume\") pod \"payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj\" (UID: \"41835b87-f8c5-49d1-8a70-1ce7bf105487\") " pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:46.491628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.491513 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66r9s\" (UniqueName: \"kubernetes.io/projected/41835b87-f8c5-49d1-8a70-1ce7bf105487-kube-api-access-66r9s\") pod \"payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj\" (UID: \"41835b87-f8c5-49d1-8a70-1ce7bf105487\") " pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:46.569167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.569138 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:46.949166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:46.949137 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj"] Oct 31 00:50:46.950028 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:50:46.949990 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41835b87_f8c5_49d1_8a70_1ce7bf105487.slice/crio-ce97391023f558fbc53a0a6fb53dbb3340963fc36616904dd54cb47b36e54a5d WatchSource:0}: Error finding container ce97391023f558fbc53a0a6fb53dbb3340963fc36616904dd54cb47b36e54a5d: Status 404 returned error can't find the container with id ce97391023f558fbc53a0a6fb53dbb3340963fc36616904dd54cb47b36e54a5d Oct 31 00:50:47.477653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:47.477609 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" event={"ID":"41835b87-f8c5-49d1-8a70-1ce7bf105487","Type":"ContainerStarted","Data":"556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f"} Oct 31 00:50:47.478058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:47.477668 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" event={"ID":"41835b87-f8c5-49d1-8a70-1ce7bf105487","Type":"ContainerStarted","Data":"ce97391023f558fbc53a0a6fb53dbb3340963fc36616904dd54cb47b36e54a5d"} Oct 31 00:50:47.478058 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:47.477760 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:47.593926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:47.593877 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" podStartSLOduration=1.593863384 podStartE2EDuration="1.593863384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:47.592270944 +0000 UTC m=+2880.067629366" watchObservedRunningTime="2026-10-31 00:50:47.593863384 +0000 UTC m=+2880.069221834" Oct 31 00:50:58.488359 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:58.488336 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:50:58.580424 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:58.580392 4969 generic.go:356] "Generic (PLEG): container finished" podID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerID="52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447" exitCode=0 Oct 31 00:50:58.580614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:58.580463 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" event={"ID":"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44","Type":"ContainerDied","Data":"52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447"} Oct 31 00:50:58.581017 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:58.580998 4969 scope.go:122] "RemoveContainer" containerID="52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447" Oct 31 00:50:58.582785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:58.582754 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:58.582898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:58.582824 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:58.582898 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:58.582854 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:59.608994 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:50:59.608960 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" event={"ID":"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44","Type":"ContainerStarted","Data":"9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45"} Oct 31 00:51:00.104422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:00.104383 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:51:00.105909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:00.105880 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 00:51:00.106037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:00.105937 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 00:51:00.874268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:00.874239 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:51:01.105163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:01.105127 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 00:51:01.105350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:01.105193 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 00:51:02.105071 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:02.105032 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 00:51:02.105533 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:02.105102 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 00:51:02.730652 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:51:02.730584 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:51:03.105650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:03.105525 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 00:51:03.106274 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:03.106239 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 00:51:04.104893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:04.104860 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 31 00:51:04.105089 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:04.104929 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 31 00:51:05.109891 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:05.109858 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:51:05.111100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:05.111082 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 00:51:15.619725 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.619683 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs"] Oct 31 00:51:15.632403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.632370 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.674939 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.674895 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs"] Oct 31 00:51:15.738183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.738135 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-4743f8-serving-cert\"" Oct 31 00:51:15.744550 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.744521 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.744944 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.744920 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.745100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.745083 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7807fb24-677e-40cd-baa5-e0d023dd47dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.745234 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.745220 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4z6w\" (UniqueName: \"kubernetes.io/projected/7807fb24-677e-40cd-baa5-e0d023dd47dc-kube-api-access-d4z6w\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.847414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.846992 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4z6w\" (UniqueName: \"kubernetes.io/projected/7807fb24-677e-40cd-baa5-e0d023dd47dc-kube-api-access-d4z6w\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.847414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.847142 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.847414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.847215 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.847414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.847241 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7807fb24-677e-40cd-baa5-e0d023dd47dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.847414 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:51:15.847355 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-serving-cert: secret "maas-api-e2e-shared-a-w4-4743f8-serving-cert" not found Oct 31 00:51:15.847784 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:51:15.847430 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls podName:7807fb24-677e-40cd-baa5-e0d023dd47dc nodeName:}" failed. No retries permitted until 2026-10-31 00:51:16.347411493 +0000 UTC m=+2908.822769893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls") pod "maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" (UID: "7807fb24-677e-40cd-baa5-e0d023dd47dc") : secret "maas-api-e2e-shared-a-w4-4743f8-serving-cert" not found Oct 31 00:51:15.848403 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.848379 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7807fb24-677e-40cd-baa5-e0d023dd47dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.850922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.850895 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:15.924142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:15.923460 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4z6w\" (UniqueName: \"kubernetes.io/projected/7807fb24-677e-40cd-baa5-e0d023dd47dc-kube-api-access-d4z6w\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:16.360941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:16.360856 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:16.363730 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:16.363710 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:16.546382 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:16.546348 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:16.971438 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:51:16.971407 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7807fb24_677e_40cd_baa5_e0d023dd47dc.slice/crio-c2ea7d7f1b0098755c0869ed4207aab70bec0f2d6be43eceb86bc205c36780bf WatchSource:0}: Error finding container c2ea7d7f1b0098755c0869ed4207aab70bec0f2d6be43eceb86bc205c36780bf: Status 404 returned error can't find the container with id c2ea7d7f1b0098755c0869ed4207aab70bec0f2d6be43eceb86bc205c36780bf Oct 31 00:51:16.972893 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:16.972868 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs"] Oct 31 00:51:17.770080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:17.770045 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" event={"ID":"7807fb24-677e-40cd-baa5-e0d023dd47dc","Type":"ContainerStarted","Data":"c2ea7d7f1b0098755c0869ed4207aab70bec0f2d6be43eceb86bc205c36780bf"} Oct 31 00:51:18.806677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:18.803339 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" event={"ID":"7807fb24-677e-40cd-baa5-e0d023dd47dc","Type":"ContainerStarted","Data":"9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279"} Oct 31 00:51:18.806677 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:18.803438 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:18.924746 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:18.924695 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" podStartSLOduration=3.037334976 podStartE2EDuration="3.924679668s" totalImagesPullingTime="887.344692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:18.919486682 +0000 UTC m=+2911.394845105" watchObservedRunningTime="2026-10-31 00:51:18.924679668 +0000 UTC m=+2911.400038090" Oct 31 00:51:19.150765 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:19.150714 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:51:19.150978 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:19.150798 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:51:22.862338 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:22.862303 4969 generic.go:356] "Generic (PLEG): container finished" podID="7697441d-50ec-428c-90ef-1286c6498dc6" containerID="98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6" exitCode=0 Oct 31 00:51:22.862855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:22.862374 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" event={"ID":"7697441d-50ec-428c-90ef-1286c6498dc6","Type":"ContainerDied","Data":"98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6"} Oct 31 00:51:22.862855 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:22.862787 4969 scope.go:122] "RemoveContainer" containerID="98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6" Oct 31 00:51:22.864236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:22.864212 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:51:22.864317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:22.864294 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:51:22.864364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:22.864329 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:51:23.872688 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:23.872654 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" event={"ID":"7697441d-50ec-428c-90ef-1286c6498dc6","Type":"ContainerStarted","Data":"bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce"} Oct 31 00:51:24.061235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.061198 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:51:24.061420 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.061401 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:51:24.061466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.061451 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 00:51:24.841065 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.840983 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 00:51:24.900943 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.900911 4969 generic.go:356] "Generic (PLEG): container finished" podID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerID="efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845" exitCode=0 Oct 31 00:51:24.901791 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.901666 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" event={"ID":"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0","Type":"ContainerDied","Data":"efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845"} Oct 31 00:51:24.902389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.902363 4969 scope.go:122] "RemoveContainer" containerID="efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845" Oct 31 00:51:24.904055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.904027 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:51:24.904151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.904098 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:51:24.904151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:24.904131 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:51:25.061397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:25.061358 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:51:25.061577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:25.061463 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 00:51:25.931733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:25.931673 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" event={"ID":"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0","Type":"ContainerStarted","Data":"4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64"} Oct 31 00:51:26.061007 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:26.060971 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:51:26.061208 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:26.061021 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 00:51:26.921986 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:26.921948 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:51:26.922259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:26.922074 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 00:51:26.922259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:26.922120 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 00:51:27.060657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:27.060627 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:51:27.061075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:27.060672 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 00:51:27.921566 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:27.921523 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 00:51:27.921890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:27.921855 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 00:51:28.061159 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:28.061123 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:51:28.061577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:28.061196 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 00:51:28.926080 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:28.926038 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 00:51:28.926401 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:28.926372 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 00:51:29.061214 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:29.061178 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Oct 31 00:51:29.061637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:29.061248 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Oct 31 00:51:29.920708 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:29.920669 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 00:51:29.920906 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:29.920740 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 00:51:30.065987 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:30.065957 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:51:30.066463 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:30.066445 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:51:30.067090 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:30.067071 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 00:51:30.924618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:30.921251 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 31 00:51:30.924618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:30.921311 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 31 00:51:31.332097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.332015 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj"] Oct 31 00:51:31.337185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.337158 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.416884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.416840 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-1a1181-serving-cert\"" Oct 31 00:51:31.418364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.418339 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj"] Oct 31 00:51:31.473630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.470519 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-metrics-certs\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.473630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.470740 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079f029a-9241-4726-b35a-001cfba9d5d6-openshift-service-ca\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.473630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.470823 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99qb\" (UniqueName: \"kubernetes.io/projected/079f029a-9241-4726-b35a-001cfba9d5d6-kube-api-access-p99qb\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.473630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.470948 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.574094 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.574025 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-metrics-certs\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.574094 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.574097 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079f029a-9241-4726-b35a-001cfba9d5d6-openshift-service-ca\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.574540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.574160 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99qb\" (UniqueName: \"kubernetes.io/projected/079f029a-9241-4726-b35a-001cfba9d5d6-kube-api-access-p99qb\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.574540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.574217 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.574540 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:51:31.574487 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-serving-cert: secret "maas-api-e2e-autores-1a1181-serving-cert" not found Oct 31 00:51:31.575198 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:51:31.574578 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls podName:079f029a-9241-4726-b35a-001cfba9d5d6 nodeName:}" failed. No retries permitted until 2026-10-31 00:51:32.074553121 +0000 UTC m=+2924.549911537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls") pod "maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" (UID: "079f029a-9241-4726-b35a-001cfba9d5d6") : secret "maas-api-e2e-autores-1a1181-serving-cert" not found Oct 31 00:51:31.599678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.599473 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079f029a-9241-4726-b35a-001cfba9d5d6-openshift-service-ca\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.632203 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.632172 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-metrics-certs\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.708304 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.708263 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99qb\" (UniqueName: \"kubernetes.io/projected/079f029a-9241-4726-b35a-001cfba9d5d6-kube-api-access-p99qb\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:31.925203 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.925170 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:51:31.925674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.925654 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:51:31.926349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:31.926334 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 00:51:32.103753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:32.103714 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:32.106981 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:32.106954 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls\") pod \"maas-api-e2e-autores-1a1181-6784596cd6-6tfvj\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:32.251172 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:32.251095 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:32.663172 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:32.663137 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj"] Oct 31 00:51:32.663574 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:51:32.663386 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079f029a_9241_4726_b35a_001cfba9d5d6.slice/crio-a5e33d721c38554c9b2f9cac3bff2d5cc884436ee705c4e01d8ee14a6beca602 WatchSource:0}: Error finding container a5e33d721c38554c9b2f9cac3bff2d5cc884436ee705c4e01d8ee14a6beca602: Status 404 returned error can't find the container with id a5e33d721c38554c9b2f9cac3bff2d5cc884436ee705c4e01d8ee14a6beca602 Oct 31 00:51:32.732069 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:51:32.732033 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:51:33.013656 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:33.013516 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" event={"ID":"079f029a-9241-4726-b35a-001cfba9d5d6","Type":"ContainerStarted","Data":"a5e33d721c38554c9b2f9cac3bff2d5cc884436ee705c4e01d8ee14a6beca602"} Oct 31 00:51:34.022149 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:34.022058 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" event={"ID":"079f029a-9241-4726-b35a-001cfba9d5d6","Type":"ContainerStarted","Data":"abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08"} Oct 31 00:51:34.022149 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:34.022119 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:34.140744 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:34.140696 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" podStartSLOduration=2.16177591 podStartE2EDuration="3.140681545s" totalImagesPullingTime="978.905635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:34.138953135 +0000 UTC m=+2926.614311559" watchObservedRunningTime="2026-10-31 00:51:34.140681545 +0000 UTC m=+2926.616039967" Oct 31 00:51:40.035578 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:40.035545 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:51:40.075182 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:40.075153 4969 generic.go:356] "Generic (PLEG): container finished" podID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerID="a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe" exitCode=0 Oct 31 00:51:40.075374 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:40.075230 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" event={"ID":"0dcd577f-e015-4cbc-a379-d8ba5fe22385","Type":"ContainerDied","Data":"a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe"} Oct 31 00:51:40.075655 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:40.075636 4969 scope.go:122] "RemoveContainer" containerID="a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe" Oct 31 00:51:40.077213 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:40.077180 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:51:40.077281 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:40.077249 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:51:40.077366 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:40.077283 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:51:41.106397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:41.106351 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" event={"ID":"0dcd577f-e015-4cbc-a379-d8ba5fe22385","Type":"ContainerStarted","Data":"938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc"} Oct 31 00:51:41.143117 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:41.143086 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:51:41.144558 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:41.144533 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 31 00:51:41.144697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:41.144578 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 31 00:51:42.144032 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:42.143999 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 31 00:51:42.144446 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:42.144049 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 31 00:51:43.143609 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:43.143571 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 31 00:51:43.143798 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:43.143661 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 31 00:51:44.143885 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:44.143852 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 31 00:51:44.144290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:44.143927 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 31 00:51:44.348777 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:44.348738 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:51:45.143802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:45.143768 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 31 00:51:45.144005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:45.143828 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 31 00:51:46.143817 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:46.143778 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 31 00:51:46.144024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:46.143838 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 31 00:51:47.149698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:47.149669 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:51:47.150853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:47.150831 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 00:51:49.626817 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:49.626787 4969 scope.go:122] "RemoveContainer" containerID="97dc8ec6ab54250a8795d16b7cfae23a00eed2193fe6abce399c0906da02c579" Oct 31 00:51:56.482617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.472353 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8"] Oct 31 00:51:56.504166 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.499243 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:51:56.540504 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.540473 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8"] Oct 31 00:51:56.587926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.587894 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-bf325c-dockercfg-xz6gp\"" Oct 31 00:51:56.588263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.588204 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-bf325c\"" Oct 31 00:51:56.652539 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.652504 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e802b0-f574-4e5d-b253-c889fedc1716-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8\" (UID: \"84e802b0-f574-4e5d-b253-c889fedc1716\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:51:56.652539 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.652545 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsh76\" (UniqueName: \"kubernetes.io/projected/84e802b0-f574-4e5d-b253-c889fedc1716-kube-api-access-rsh76\") pod \"payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8\" (UID: \"84e802b0-f574-4e5d-b253-c889fedc1716\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:51:56.753756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.753663 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e802b0-f574-4e5d-b253-c889fedc1716-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8\" (UID: \"84e802b0-f574-4e5d-b253-c889fedc1716\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:51:56.753756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.753704 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsh76\" (UniqueName: \"kubernetes.io/projected/84e802b0-f574-4e5d-b253-c889fedc1716-kube-api-access-rsh76\") pod \"payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8\" (UID: \"84e802b0-f574-4e5d-b253-c889fedc1716\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:51:56.754462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.754434 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e802b0-f574-4e5d-b253-c889fedc1716-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8\" (UID: \"84e802b0-f574-4e5d-b253-c889fedc1716\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:51:56.815203 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.815173 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsh76\" (UniqueName: \"kubernetes.io/projected/84e802b0-f574-4e5d-b253-c889fedc1716-kube-api-access-rsh76\") pod \"payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8\" (UID: \"84e802b0-f574-4e5d-b253-c889fedc1716\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:51:56.840953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:56.840912 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:51:57.248010 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:51:57.247981 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e802b0_f574_4e5d_b253_c889fedc1716.slice/crio-d14694cce8cab3cea4a4b7b441a338c34293bf8ed7f314ed44ba2f2dabd9357e WatchSource:0}: Error finding container d14694cce8cab3cea4a4b7b441a338c34293bf8ed7f314ed44ba2f2dabd9357e: Status 404 returned error can't find the container with id d14694cce8cab3cea4a4b7b441a338c34293bf8ed7f314ed44ba2f2dabd9357e Oct 31 00:51:57.250674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:57.250650 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8"] Oct 31 00:51:58.250062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:58.250027 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" event={"ID":"84e802b0-f574-4e5d-b253-c889fedc1716","Type":"ContainerStarted","Data":"b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807"} Oct 31 00:51:58.250521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:58.250107 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" event={"ID":"84e802b0-f574-4e5d-b253-c889fedc1716","Type":"ContainerStarted","Data":"d14694cce8cab3cea4a4b7b441a338c34293bf8ed7f314ed44ba2f2dabd9357e"} Oct 31 00:51:58.250521 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:58.250259 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:51:58.369562 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:51:58.369509 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" podStartSLOduration=2.369493095 podStartE2EDuration="2.369493095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:58.365644542 +0000 UTC m=+2950.841002966" watchObservedRunningTime="2026-10-31 00:51:58.369493095 +0000 UTC m=+2950.844851553" Oct 31 00:52:02.730090 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:52:02.730049 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:52:05.307535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:05.307500 4969 generic.go:356] "Generic (PLEG): container finished" podID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerID="e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9" exitCode=0 Oct 31 00:52:05.308012 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:05.307572 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" event={"ID":"e7c58d5b-e672-4e93-8bfe-4f7677c31125","Type":"ContainerDied","Data":"e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9"} Oct 31 00:52:05.308062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:05.308013 4969 scope.go:122] "RemoveContainer" containerID="e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9" Oct 31 00:52:05.309553 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:05.309528 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:52:05.309633 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:05.309612 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:52:05.309683 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:05.309647 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:52:06.320319 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:06.320280 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" event={"ID":"e7c58d5b-e672-4e93-8bfe-4f7677c31125","Type":"ContainerStarted","Data":"ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5"} Oct 31 00:52:06.375032 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:06.374999 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:52:06.376456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:06.376429 4969 patch_prober.go:29] interesting pod/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:52:06.376627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:06.376476 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 31 00:52:07.375674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:07.375639 4969 patch_prober.go:29] interesting pod/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:52:07.376070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:07.375691 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 31 00:52:08.374943 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:08.374905 4969 patch_prober.go:29] interesting pod/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:52:08.375129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:08.374965 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 31 00:52:09.261259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:09.261231 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 00:52:09.375140 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:09.375102 4969 patch_prober.go:29] interesting pod/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:52:09.375315 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:09.375157 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 31 00:52:10.375392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:10.375360 4969 patch_prober.go:29] interesting pod/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:52:10.375941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:10.375418 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 31 00:52:10.425899 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:10.425869 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:52:11.375674 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:11.375643 4969 patch_prober.go:29] interesting pod/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 31 00:52:11.376107 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:11.375698 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 31 00:52:12.380029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:12.379998 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:52:12.381349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:12.381322 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:52:16.232131 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.232093 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57"] Oct 31 00:52:16.236805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.236781 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.292129 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.292078 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-c6b3b9-serving-cert\"" Oct 31 00:52:16.296220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.296193 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57"] Oct 31 00:52:16.328124 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.328090 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.328317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.328294 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.328397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.328371 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.328473 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.328457 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq76n\" (UniqueName: \"kubernetes.io/projected/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-kube-api-access-dq76n\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.429927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.429884 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.430105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.429948 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.430105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.430015 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq76n\" (UniqueName: \"kubernetes.io/projected/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-kube-api-access-dq76n\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.430265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.430116 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.430265 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:52:16.430248 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-serving-cert: secret "maas-api-e2e-shared-b-w0-c6b3b9-serving-cert" not found Oct 31 00:52:16.430378 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:52:16.430316 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls podName:902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f nodeName:}" failed. No retries permitted until 2026-10-31 00:52:16.930294009 +0000 UTC m=+2969.405652424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls") pod "maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" (UID: "902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f") : secret "maas-api-e2e-shared-b-w0-c6b3b9-serving-cert" not found Oct 31 00:52:16.430785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.430766 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.433113 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.433086 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.491705 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.491615 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq76n\" (UniqueName: \"kubernetes.io/projected/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-kube-api-access-dq76n\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.936294 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.936246 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:16.939043 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:16.939020 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:17.153231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:17.153197 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:17.535187 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:17.535157 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57"] Oct 31 00:52:17.535664 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:52:17.535382 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902e7d5e_0dc8_40ac_b8c8_43ae0105ba3f.slice/crio-dc9793ee3b93d5986c4879ecc0eda82f53c0eebcf9b39c7c9b6c59b4d3b6a001 WatchSource:0}: Error finding container dc9793ee3b93d5986c4879ecc0eda82f53c0eebcf9b39c7c9b6c59b4d3b6a001: Status 404 returned error can't find the container with id dc9793ee3b93d5986c4879ecc0eda82f53c0eebcf9b39c7c9b6c59b4d3b6a001 Oct 31 00:52:18.412000 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:18.411963 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" event={"ID":"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f","Type":"ContainerStarted","Data":"dc9793ee3b93d5986c4879ecc0eda82f53c0eebcf9b39c7c9b6c59b4d3b6a001"} Oct 31 00:52:19.423186 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:19.423146 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" event={"ID":"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f","Type":"ContainerStarted","Data":"7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6"} Oct 31 00:52:19.443268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:19.443235 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:19.546369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:19.546318 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" podStartSLOduration=2.667545192 podStartE2EDuration="3.546298979s" totalImagesPullingTime="878.753787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:19.541882097 +0000 UTC m=+2972.017240518" watchObservedRunningTime="2026-10-31 00:52:19.546298979 +0000 UTC m=+2972.021657403" Oct 31 00:52:26.446059 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.446031 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 00:52:26.552227 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.552198 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727"] Oct 31 00:52:26.556627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.556610 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.608737 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.608703 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-4e1a6c-serving-cert\"" Oct 31 00:52:26.608952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.608745 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727"] Oct 31 00:52:26.649158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.649120 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.649322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.649171 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfqd\" (UniqueName: \"kubernetes.io/projected/1558a8d2-5076-43b8-8817-fdead1d9ada7-kube-api-access-2kfqd\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.649322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.649264 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1558a8d2-5076-43b8-8817-fdead1d9ada7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.649462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.649442 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.750220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.750139 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1558a8d2-5076-43b8-8817-fdead1d9ada7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.750392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.750239 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.750392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.750380 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.750528 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:52:26.750385 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-serving-cert: secret "maas-api-e2e-shared-b-w3-4e1a6c-serving-cert" not found Oct 31 00:52:26.750528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.750418 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfqd\" (UniqueName: \"kubernetes.io/projected/1558a8d2-5076-43b8-8817-fdead1d9ada7-kube-api-access-2kfqd\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.750528 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:52:26.750453 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls podName:1558a8d2-5076-43b8-8817-fdead1d9ada7 nodeName:}" failed. No retries permitted until 2026-10-31 00:52:27.250432289 +0000 UTC m=+2979.725790704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls") pod "maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" (UID: "1558a8d2-5076-43b8-8817-fdead1d9ada7") : secret "maas-api-e2e-shared-b-w3-4e1a6c-serving-cert" not found Oct 31 00:52:26.750875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.750857 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1558a8d2-5076-43b8-8817-fdead1d9ada7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.753013 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.752996 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:26.811657 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:26.811624 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfqd\" (UniqueName: \"kubernetes.io/projected/1558a8d2-5076-43b8-8817-fdead1d9ada7-kube-api-access-2kfqd\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:27.257748 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:27.257716 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:27.260378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:27.260350 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:27.471000 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:27.470968 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:27.854198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:27.854168 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727"] Oct 31 00:52:27.855058 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:52:27.855030 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1558a8d2_5076_43b8_8817_fdead1d9ada7.slice/crio-7694f109d94ef86a70b604ecc396e23fb1355994010aed7514f877313852a53b WatchSource:0}: Error finding container 7694f109d94ef86a70b604ecc396e23fb1355994010aed7514f877313852a53b: Status 404 returned error can't find the container with id 7694f109d94ef86a70b604ecc396e23fb1355994010aed7514f877313852a53b Oct 31 00:52:28.496472 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:28.496435 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" event={"ID":"1558a8d2-5076-43b8-8817-fdead1d9ada7","Type":"ContainerStarted","Data":"7694f109d94ef86a70b604ecc396e23fb1355994010aed7514f877313852a53b"} Oct 31 00:52:29.505195 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:29.505153 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" event={"ID":"1558a8d2-5076-43b8-8817-fdead1d9ada7","Type":"ContainerStarted","Data":"5c73757958c0b82200dc1aee5438bdaf5bac95b56863390a3758431e3726de35"} Oct 31 00:52:29.505681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:29.505224 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:29.625838 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:29.625786 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" podStartSLOduration=2.741467824 podStartE2EDuration="3.625772161s" totalImagesPullingTime="884.304337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:29.622078487 +0000 UTC m=+2982.097436911" watchObservedRunningTime="2026-10-31 00:52:29.625772161 +0000 UTC m=+2982.101130585" Oct 31 00:52:32.727621 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:52:32.727557 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:52:35.518889 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:35.518861 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 00:52:48.271806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:48.271776 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:52:48.288681 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:48.288654 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:52:48.290268 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:48.290248 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:52:48.304145 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:52:48.304121 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:53:02.728512 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:53:02.728427 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:53:32.728765 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:53:32.728731 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:53:50.959872 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:50.959816 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g"] Oct 31 00:53:50.964723 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:50.964697 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.010583 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.010545 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tls-certs\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.010810 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.010664 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbjn\" (UniqueName: \"kubernetes.io/projected/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-kube-api-access-grbjn\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.010886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.010841 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-home\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.010946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.010886 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-dshm\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.010946 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.010910 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-model-cache\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.011061 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.011038 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tmp-dir\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.013835 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.013808 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g"] Oct 31 00:53:51.058634 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.058607 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-cfc055\"/\"default-dockercfg-gnfl8\"" Oct 31 00:53:51.058790 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.058640 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-cfc055\"/\"openshift-service-ca.crt\"" Oct 31 00:53:51.066243 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.066216 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-cfc055\"/\"kube-root-ca.crt\"" Oct 31 00:53:51.066355 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.066298 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-cfc055\"/\"auth-test-model-cfc055-kserve-self-signed-certs\"" Oct 31 00:53:51.112199 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.112165 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tmp-dir\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.114516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.112259 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tls-certs\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.114516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.112302 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbjn\" (UniqueName: \"kubernetes.io/projected/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-kube-api-access-grbjn\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.114516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.112353 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-home\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.114516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.112375 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-dshm\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.114516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.112389 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-model-cache\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.114516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.112801 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-home\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.114516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.112932 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-model-cache\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.114516 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.113983 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tmp-dir\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.116938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.116910 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-dshm\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.120694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.120653 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tls-certs\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.176395 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.176365 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbjn\" (UniqueName: \"kubernetes.io/projected/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-kube-api-access-grbjn\") pod \"auth-test-model-cfc055-kserve-5dd8596657-zwf8g\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.279628 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.279513 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:51.659339 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:51.659306 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g"] Oct 31 00:53:51.659546 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:53:51.659521 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddedc0b00_d8de_4b79_8b1d_80f3dda6c964.slice/crio-7ae8329fb18c09d8ae043e9890ef18eb8f1c25ea2d123c478b171274339bed49 WatchSource:0}: Error finding container 7ae8329fb18c09d8ae043e9890ef18eb8f1c25ea2d123c478b171274339bed49: Status 404 returned error can't find the container with id 7ae8329fb18c09d8ae043e9890ef18eb8f1c25ea2d123c478b171274339bed49 Oct 31 00:53:52.168797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:52.168760 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" event={"ID":"dedc0b00-d8de-4b79-8b1d-80f3dda6c964","Type":"ContainerStarted","Data":"182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86"} Oct 31 00:53:52.169216 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:52.168806 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" event={"ID":"dedc0b00-d8de-4b79-8b1d-80f3dda6c964","Type":"ContainerStarted","Data":"7ae8329fb18c09d8ae043e9890ef18eb8f1c25ea2d123c478b171274339bed49"} Oct 31 00:53:52.280509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:52.280474 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:53:52.289797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:52.289754 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" podStartSLOduration=2.289740089 podStartE2EDuration="2.289740089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:52.285744245 +0000 UTC m=+3064.761102669" watchObservedRunningTime="2026-10-31 00:53:52.289740089 +0000 UTC m=+3064.765098511" Oct 31 00:53:59.843095 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:53:59.843063 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj"] Oct 31 00:54:01.279897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:01.279861 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:54:01.294767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:01.294735 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:54:02.259151 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:02.259122 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:54:02.729745 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:54:02.729706 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:54:04.844430 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:04.844389 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" podUID="41835b87-f8c5-49d1-8a70-1ce7bf105487" containerName="payload-processing" containerID="cri-o://556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f" gracePeriod=25 Oct 31 00:54:05.115073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.115039 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:54:05.192759 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.192724 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41835b87-f8c5-49d1-8a70-1ce7bf105487-kube-api-access-66r9s\" (UniqueName: \"kubernetes.io/projected/41835b87-f8c5-49d1-8a70-1ce7bf105487-kube-api-access-66r9s\") pod \"41835b87-f8c5-49d1-8a70-1ce7bf105487\" (UID: \"41835b87-f8c5-49d1-8a70-1ce7bf105487\") " Oct 31 00:54:05.192958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.192808 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41835b87-f8c5-49d1-8a70-1ce7bf105487-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41835b87-f8c5-49d1-8a70-1ce7bf105487-plugins-config-volume\") pod \"41835b87-f8c5-49d1-8a70-1ce7bf105487\" (UID: \"41835b87-f8c5-49d1-8a70-1ce7bf105487\") " Oct 31 00:54:05.193204 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.193181 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41835b87-f8c5-49d1-8a70-1ce7bf105487-plugins-config-volume" pod "41835b87-f8c5-49d1-8a70-1ce7bf105487" (UID: "41835b87-f8c5-49d1-8a70-1ce7bf105487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:05.193350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.193329 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41835b87-f8c5-49d1-8a70-1ce7bf105487-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:54:05.195280 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.195256 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41835b87-f8c5-49d1-8a70-1ce7bf105487-kube-api-access-66r9s" pod "41835b87-f8c5-49d1-8a70-1ce7bf105487" (UID: "41835b87-f8c5-49d1-8a70-1ce7bf105487"). InnerVolumeSpecName "kube-api-access-66r9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:05.276507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.276472 4969 generic.go:356] "Generic (PLEG): container finished" podID="41835b87-f8c5-49d1-8a70-1ce7bf105487" containerID="556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f" exitCode=0 Oct 31 00:54:05.276769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.276523 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" Oct 31 00:54:05.276769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.276550 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" event={"ID":"41835b87-f8c5-49d1-8a70-1ce7bf105487","Type":"ContainerDied","Data":"556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f"} Oct 31 00:54:05.276769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.276648 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj" event={"ID":"41835b87-f8c5-49d1-8a70-1ce7bf105487","Type":"ContainerDied","Data":"ce97391023f558fbc53a0a6fb53dbb3340963fc36616904dd54cb47b36e54a5d"} Oct 31 00:54:05.276769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.276673 4969 scope.go:122] "RemoveContainer" containerID="556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f" Oct 31 00:54:05.288862 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.288836 4969 scope.go:122] "RemoveContainer" containerID="556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f" Oct 31 00:54:05.289125 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:54:05.289107 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f\": container with ID starting with 556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f not found: ID does not exist" containerID="556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f" Oct 31 00:54:05.289178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.289133 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f"} err="failed to get container status \"556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f\": rpc error: code = NotFound desc = could not find container \"556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f\": container with ID starting with 556aceb429e86122291362e07fc8cfbe803d5f2fc090cb6686de499ef056240f not found: ID does not exist" Oct 31 00:54:05.294144 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.294124 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66r9s\" (UniqueName: \"kubernetes.io/projected/41835b87-f8c5-49d1-8a70-1ce7bf105487-kube-api-access-66r9s\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:54:05.443021 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.442940 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj"] Oct 31 00:54:05.447057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.447033 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1a1181-6b6b84fcc9-zvgrj"] Oct 31 00:54:05.735750 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:05.735671 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41835b87-f8c5-49d1-8a70-1ce7bf105487" path="/var/lib/kubelet/pods/41835b87-f8c5-49d1-8a70-1ce7bf105487/volumes" Oct 31 00:54:14.052437 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:14.052402 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252"] Oct 31 00:54:14.102070 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:14.102041 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91841e0d-sn252"] Oct 31 00:54:15.735155 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:15.735116 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe6f9a9-967e-4401-872b-b510e15591e2" path="/var/lib/kubelet/pods/5fe6f9a9-967e-4401-872b-b510e15591e2/volumes" Oct 31 00:54:24.057632 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:24.057576 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j"] Oct 31 00:54:24.106607 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:24.106570 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd9dcfb9-smz9j"] Oct 31 00:54:25.735091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:25.735053 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443f3255-5bd9-4902-83fd-47a3bea32316" path="/var/lib/kubelet/pods/443f3255-5bd9-4902-83fd-47a3bea32316/volumes" Oct 31 00:54:32.729151 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:54:32.729112 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:54:39.064614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:39.064554 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj"] Oct 31 00:54:39.114644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:39.114586 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2"] Oct 31 00:54:39.118538 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:39.118519 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0db33ab-knsmj"] Oct 31 00:54:39.121715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:39.121696 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32d3d330-l7gk2"] Oct 31 00:54:39.736085 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:39.736052 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c148b9-60f6-4df0-a473-dd731078121d" path="/var/lib/kubelet/pods/13c148b9-60f6-4df0-a473-dd731078121d/volumes" Oct 31 00:54:39.736546 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:39.736531 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531a267e-a00f-4791-b9df-bcd217f4368c" path="/var/lib/kubelet/pods/531a267e-a00f-4791-b9df-bcd217f4368c/volumes" Oct 31 00:54:51.048936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:51.048903 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb"] Oct 31 00:54:51.098601 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:51.098571 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8f2378-22zcb"] Oct 31 00:54:51.735723 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:54:51.735693 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d82be4-6d76-450a-a186-6f3bd7c52ef4" path="/var/lib/kubelet/pods/75d82be4-6d76-450a-a186-6f3bd7c52ef4/volumes" Oct 31 00:55:02.727623 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:55:02.727563 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:55:02.869378 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:02.869346 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj"] Oct 31 00:55:02.869584 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:02.869558 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" podUID="079f029a-9241-4726-b35a-001cfba9d5d6" containerName="maas-api" containerID="cri-o://abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08" gracePeriod=30 Oct 31 00:55:03.167587 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.167555 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:55:03.229977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.229937 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079f029a-9241-4726-b35a-001cfba9d5d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079f029a-9241-4726-b35a-001cfba9d5d6-openshift-service-ca\") pod \"079f029a-9241-4726-b35a-001cfba9d5d6\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " Oct 31 00:55:03.230156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.230029 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-metrics-certs\") pod \"079f029a-9241-4726-b35a-001cfba9d5d6\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " Oct 31 00:55:03.230156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.230100 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079f029a-9241-4726-b35a-001cfba9d5d6-kube-api-access-p99qb\" (UniqueName: \"kubernetes.io/projected/079f029a-9241-4726-b35a-001cfba9d5d6-kube-api-access-p99qb\") pod \"079f029a-9241-4726-b35a-001cfba9d5d6\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " Oct 31 00:55:03.230266 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.230212 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls\") pod \"079f029a-9241-4726-b35a-001cfba9d5d6\" (UID: \"079f029a-9241-4726-b35a-001cfba9d5d6\") " Oct 31 00:55:03.230571 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.230540 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079f029a-9241-4726-b35a-001cfba9d5d6-openshift-service-ca" pod "079f029a-9241-4726-b35a-001cfba9d5d6" (UID: "079f029a-9241-4726-b35a-001cfba9d5d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:03.230963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.230907 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/079f029a-9241-4726-b35a-001cfba9d5d6-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:03.232529 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.232509 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-metrics-certs" pod "079f029a-9241-4726-b35a-001cfba9d5d6" (UID: "079f029a-9241-4726-b35a-001cfba9d5d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:03.233224 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.233183 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls" pod "079f029a-9241-4726-b35a-001cfba9d5d6" (UID: "079f029a-9241-4726-b35a-001cfba9d5d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:03.233224 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.233214 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079f029a-9241-4726-b35a-001cfba9d5d6-kube-api-access-p99qb" pod "079f029a-9241-4726-b35a-001cfba9d5d6" (UID: "079f029a-9241-4726-b35a-001cfba9d5d6"). InnerVolumeSpecName "kube-api-access-p99qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:03.332637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.332544 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:03.332637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.332611 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p99qb\" (UniqueName: \"kubernetes.io/projected/079f029a-9241-4726-b35a-001cfba9d5d6-kube-api-access-p99qb\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:03.332637 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.332630 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/079f029a-9241-4726-b35a-001cfba9d5d6-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:03.798126 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.798083 4969 generic.go:356] "Generic (PLEG): container finished" podID="079f029a-9241-4726-b35a-001cfba9d5d6" containerID="abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08" exitCode=0 Oct 31 00:55:03.798675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.798122 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" event={"ID":"079f029a-9241-4726-b35a-001cfba9d5d6","Type":"ContainerDied","Data":"abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08"} Oct 31 00:55:03.798675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.798176 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" Oct 31 00:55:03.798675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.798193 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj" event={"ID":"079f029a-9241-4726-b35a-001cfba9d5d6","Type":"ContainerDied","Data":"a5e33d721c38554c9b2f9cac3bff2d5cc884436ee705c4e01d8ee14a6beca602"} Oct 31 00:55:03.798675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.798229 4969 scope.go:122] "RemoveContainer" containerID="abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08" Oct 31 00:55:03.844665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.835940 4969 scope.go:122] "RemoveContainer" containerID="abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08" Oct 31 00:55:03.844665 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:55:03.836911 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08\": container with ID starting with abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08 not found: ID does not exist" containerID="abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08" Oct 31 00:55:03.844665 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.836952 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08"} err="failed to get container status \"abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08\": rpc error: code = NotFound desc = could not find container \"abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08\": container with ID starting with abd02b1af2ecd592b0cf2030f9c75f90f289d1c73a872444bf2a17443dbebf08 not found: ID does not exist" Oct 31 00:55:03.967630 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.967558 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj"] Oct 31 00:55:03.971295 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:03.971272 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1a1181-6784596cd6-6tfvj"] Oct 31 00:55:05.735886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:05.735855 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079f029a-9241-4726-b35a-001cfba9d5d6" path="/var/lib/kubelet/pods/079f029a-9241-4726-b35a-001cfba9d5d6/volumes" Oct 31 00:55:06.049405 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:06.049328 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7"] Oct 31 00:55:06.098808 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:06.098777 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86289c90-qvcf7"] Oct 31 00:55:07.735653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:07.735585 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4e4412-02a1-4942-9ee7-a77fd6a9133f" path="/var/lib/kubelet/pods/2e4e4412-02a1-4942-9ee7-a77fd6a9133f/volumes" Oct 31 00:55:12.305363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:12.305328 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9"] Oct 31 00:55:12.305938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:12.305546 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" containerID="cri-o://ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5" gracePeriod=30 Oct 31 00:55:12.414714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:12.414672 4969 patch_prober.go:29] interesting pod/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:55:12.415010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:12.414978 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:55:13.617761 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.617723 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd"] Oct 31 00:55:13.618544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.618521 4969 cpu_manager.go:414] "containerMap: removing container" podUID="41835b87-f8c5-49d1-8a70-1ce7bf105487" containerName="payload-processing" Oct 31 00:55:13.618714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.618546 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41835b87-f8c5-49d1-8a70-1ce7bf105487" containerName="payload-processing" Oct 31 00:55:13.618714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.618564 4969 cpu_manager.go:414] "containerMap: removing container" podUID="079f029a-9241-4726-b35a-001cfba9d5d6" containerName="maas-api" Oct 31 00:55:13.618714 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.618571 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079f029a-9241-4726-b35a-001cfba9d5d6" containerName="maas-api" Oct 31 00:55:13.618884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.618833 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="41835b87-f8c5-49d1-8a70-1ce7bf105487" containerName="payload-processing" Oct 31 00:55:13.618884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.618851 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="079f029a-9241-4726-b35a-001cfba9d5d6" containerName="maas-api" Oct 31 00:55:13.623841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.623819 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.702841 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.700991 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ee480f-openshift-default-dockercfg-nbntv\"" Oct 31 00:55:13.723907 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.723859 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-token\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.724105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.723964 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vhn\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-kube-api-access-l6vhn\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.724105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.724001 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-envoy\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.724105 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.724101 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-credential-socket\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.724300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.724146 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-podinfo\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.724300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.724172 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-data\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.724300 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.724263 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-socket\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.724471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.724298 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-certs\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.724471 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.724363 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c35be1-e5b3-4873-871d-275f05ce0dae-istiod-ca-cert\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.738029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.737993 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd"] Oct 31 00:55:13.825733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.825699 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-socket\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.825733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.825738 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-certs\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826006 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.825777 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c35be1-e5b3-4873-871d-275f05ce0dae-istiod-ca-cert\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826006 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.825805 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-token\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826006 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.825961 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vhn\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-kube-api-access-l6vhn\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.826017 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-envoy\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.826129 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-socket\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826190 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.826127 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-credential-socket\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.826160 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-certs\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.826214 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-podinfo\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826365 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.826286 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-data\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.826435 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-credential-socket\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826649 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.826619 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-data\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.826715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.826637 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c35be1-e5b3-4873-871d-275f05ce0dae-istiod-ca-cert\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.828524 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.828501 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-envoy\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.828785 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.828766 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-podinfo\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.887033 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.886966 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-token\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.889048 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.889021 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vhn\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-kube-api-access-l6vhn\") pod \"e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:13.937876 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:13.937847 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:14.375753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:14.375723 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd"] Oct 31 00:55:14.376245 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:55:14.376218 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c35be1_e5b3_4873_871d_275f05ce0dae.slice/crio-284fb82ee5f7d52ba5e46e92dca0589445c554f1b58e17cbb4a15e98f2d520e6 WatchSource:0}: Error finding container 284fb82ee5f7d52ba5e46e92dca0589445c554f1b58e17cbb4a15e98f2d520e6: Status 404 returned error can't find the container with id 284fb82ee5f7d52ba5e46e92dca0589445c554f1b58e17cbb4a15e98f2d520e6 Oct 31 00:55:14.378936 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:14.378904 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:55:14.379016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:14.378968 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:55:14.379016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:14.378996 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:55:14.892097 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:14.892059 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" event={"ID":"91c35be1-e5b3-4873-871d-275f05ce0dae","Type":"ContainerStarted","Data":"ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535"} Oct 31 00:55:14.892534 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:14.892122 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" event={"ID":"91c35be1-e5b3-4873-871d-275f05ce0dae","Type":"ContainerStarted","Data":"284fb82ee5f7d52ba5e46e92dca0589445c554f1b58e17cbb4a15e98f2d520e6"} Oct 31 00:55:14.937970 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:14.937946 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:15.021621 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:15.017499 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" podStartSLOduration=2.017472128 podStartE2EDuration="2.017472128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:15.013842081 +0000 UTC m=+3147.489200506" watchObservedRunningTime="2026-10-31 00:55:15.017472128 +0000 UTC m=+3147.492830553" Oct 31 00:55:15.944462 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:15.944433 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:16.907924 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:16.907892 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:16.908717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:16.908697 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:55:17.585055 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.584997 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:55:17.670726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.670698 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-kube-api-access-l2sjn\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-kube-api-access-l2sjn\") pod \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " Oct 31 00:55:17.670926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.670742 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istiod-ca-cert\") pod \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " Oct 31 00:55:17.670926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.670795 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-podinfo\") pod \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " Oct 31 00:55:17.670926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.670854 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-data\") pod \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " Oct 31 00:55:17.670926 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.670901 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-envoy\") pod \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " Oct 31 00:55:17.671158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671036 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-certs\") pod \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " Oct 31 00:55:17.671158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671121 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-token\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-token\") pod \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " Oct 31 00:55:17.671264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671166 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-socket\") pod \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " Oct 31 00:55:17.671264 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671199 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-credential-socket\") pod \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\" (UID: \"e7c58d5b-e672-4e93-8bfe-4f7677c31125\") " Oct 31 00:55:17.671373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671161 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-data" pod "e7c58d5b-e672-4e93-8bfe-4f7677c31125" (UID: "e7c58d5b-e672-4e93-8bfe-4f7677c31125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:17.671373 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671209 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istiod-ca-cert" pod "e7c58d5b-e672-4e93-8bfe-4f7677c31125" (UID: "e7c58d5b-e672-4e93-8bfe-4f7677c31125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:17.671650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671623 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-certs" pod "e7c58d5b-e672-4e93-8bfe-4f7677c31125" (UID: "e7c58d5b-e672-4e93-8bfe-4f7677c31125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:17.671650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671698 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-credential-socket" pod "e7c58d5b-e672-4e93-8bfe-4f7677c31125" (UID: "e7c58d5b-e672-4e93-8bfe-4f7677c31125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:17.671650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671702 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:17.671650 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671653 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-socket" pod "e7c58d5b-e672-4e93-8bfe-4f7677c31125" (UID: "e7c58d5b-e672-4e93-8bfe-4f7677c31125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:17.672003 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.671740 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:17.673306 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.673274 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-kube-api-access-l2sjn" pod "e7c58d5b-e672-4e93-8bfe-4f7677c31125" (UID: "e7c58d5b-e672-4e93-8bfe-4f7677c31125"). InnerVolumeSpecName "kube-api-access-l2sjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:17.673477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.673459 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-token" pod "e7c58d5b-e672-4e93-8bfe-4f7677c31125" (UID: "e7c58d5b-e672-4e93-8bfe-4f7677c31125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:17.673796 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.673775 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-envoy" pod "e7c58d5b-e672-4e93-8bfe-4f7677c31125" (UID: "e7c58d5b-e672-4e93-8bfe-4f7677c31125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:17.673796 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.673781 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-podinfo" pod "e7c58d5b-e672-4e93-8bfe-4f7677c31125" (UID: "e7c58d5b-e672-4e93-8bfe-4f7677c31125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:17.772615 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.772565 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:17.772778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.772622 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:17.772778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.772636 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:17.772778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.772644 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:17.772778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.772654 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c58d5b-e672-4e93-8bfe-4f7677c31125-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:17.772778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.772664 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2sjn\" (UniqueName: \"kubernetes.io/projected/e7c58d5b-e672-4e93-8bfe-4f7677c31125-kube-api-access-l2sjn\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:17.772778 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.772673 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c58d5b-e672-4e93-8bfe-4f7677c31125-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:55:17.917293 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.917256 4969 generic.go:356] "Generic (PLEG): container finished" podID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerID="ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5" exitCode=0 Oct 31 00:55:17.917506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.917311 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" Oct 31 00:55:17.917506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.917334 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" event={"ID":"e7c58d5b-e672-4e93-8bfe-4f7677c31125","Type":"ContainerDied","Data":"ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5"} Oct 31 00:55:17.917506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.917386 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9" event={"ID":"e7c58d5b-e672-4e93-8bfe-4f7677c31125","Type":"ContainerDied","Data":"84cc014cb1fb3468ad7f9d2427f00704f1572f23a22be91b4f6d1bca1fb93404"} Oct 31 00:55:17.917506 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.917399 4969 scope.go:122] "RemoveContainer" containerID="ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5" Oct 31 00:55:17.928978 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.928962 4969 scope.go:122] "RemoveContainer" containerID="e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9" Oct 31 00:55:17.938977 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.938945 4969 scope.go:122] "RemoveContainer" containerID="ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5" Oct 31 00:55:17.939267 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:55:17.939248 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5\": container with ID starting with ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5 not found: ID does not exist" containerID="ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5" Oct 31 00:55:17.939331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.939274 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5"} err="failed to get container status \"ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5\": rpc error: code = NotFound desc = could not find container \"ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5\": container with ID starting with ce6c8581679373e75bd16a5b5b4e71f258174533815eefb502cfa83bc19b3af5 not found: ID does not exist" Oct 31 00:55:17.939331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.939293 4969 scope.go:122] "RemoveContainer" containerID="e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9" Oct 31 00:55:17.939513 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:55:17.939497 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9\": container with ID starting with e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9 not found: ID does not exist" containerID="e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9" Oct 31 00:55:17.939556 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:17.939518 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9"} err="failed to get container status \"e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9\": rpc error: code = NotFound desc = could not find container \"e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9\": container with ID starting with e94d444a4279f2cbb1006266cfea6ec46b3e02bd8e8172336436f70b45a9ccd9 not found: ID does not exist" Oct 31 00:55:18.049102 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:18.049067 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4"] Oct 31 00:55:18.075726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:18.075680 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91046e21-mmhr4"] Oct 31 00:55:18.083727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:18.083702 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9"] Oct 31 00:55:18.098555 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:18.098527 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-1a1181-openshift-default-7996f77b8f-djtr9"] Oct 31 00:55:19.742915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:19.742871 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2e0dc7-3bca-4f79-9e03-d638f29e605e" path="/var/lib/kubelet/pods/2e2e0dc7-3bca-4f79-9e03-d638f29e605e/volumes" Oct 31 00:55:19.743728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:19.743701 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" path="/var/lib/kubelet/pods/e7c58d5b-e672-4e93-8bfe-4f7677c31125/volumes" Oct 31 00:55:32.729152 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:55:32.729119 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:55:49.742014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:49.741977 4969 scope.go:122] "RemoveContainer" containerID="7d7994965e6398c6bc2d4ed31d98d4771eee5cf311bee1342c38f25de400b2ca" Oct 31 00:55:49.752426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:49.752404 4969 scope.go:122] "RemoveContainer" containerID="d2fe078e2bd57fa53f8a3dbba55b788fc08fd4d893d8dcd7322cdc45e1af12d0" Oct 31 00:55:49.762150 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:49.762131 4969 scope.go:122] "RemoveContainer" containerID="67210ec5337b6f3b4ab09364e69e1e25eb14d537bfb3bf562934f2ef2e6ae85b" Oct 31 00:55:49.771565 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:49.771545 4969 scope.go:122] "RemoveContainer" containerID="b847785d7d32edc276d7fc70d99c28a199a979126e263afd25e897075b37bfba" Oct 31 00:55:49.781466 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:49.781446 4969 scope.go:122] "RemoveContainer" containerID="9f5d1b10b92d33939b2eca44852c52d95a7961058d34aa42f96ca78acda557c5" Oct 31 00:55:59.742799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:59.742711 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc"] Oct 31 00:55:59.743273 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:59.742958 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" containerID="cri-o://65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a" gracePeriod=30 Oct 31 00:55:59.788489 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:59.788445 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d"] Oct 31 00:55:59.788795 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:59.788743 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" containerID="cri-o://3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52" gracePeriod=30 Oct 31 00:55:59.792051 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:59.792028 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6"] Oct 31 00:55:59.792252 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:59.792217 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" containerID="cri-o://6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4" gracePeriod=30 Oct 31 00:55:59.840027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:59.839987 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv"] Oct 31 00:55:59.840240 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:55:59.840212 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" containerID="cri-o://11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e" gracePeriod=30 Oct 31 00:56:00.701118 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.701085 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws"] Oct 31 00:56:00.701806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.701785 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" Oct 31 00:56:00.701806 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.701807 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" Oct 31 00:56:00.701989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.701970 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" Oct 31 00:56:00.702052 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.701989 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" Oct 31 00:56:00.707238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.707216 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.759958 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.759917 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4743f8\"/\"kube-root-ca.crt\"" Oct 31 00:56:00.760425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.760223 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4743f8\"/\"default-dockercfg-6x7z7\"" Oct 31 00:56:00.760425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.760256 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4743f8\"/\"openshift-service-ca.crt\"" Oct 31 00:56:00.760425 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.760295 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4743f8\"/\"auth-test-model-4743f8-kserve-self-signed-certs\"" Oct 31 00:56:00.764720 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.764692 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws"] Oct 31 00:56:00.798389 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.798358 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778e00fa-3daa-47b9-866e-4c742d50333b-tls-certs\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.798568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.798451 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-tmp-dir\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.798568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.798485 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-home\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.798568 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.798512 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-model-cache\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.798794 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.798611 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjrg\" (UniqueName: \"kubernetes.io/projected/778e00fa-3daa-47b9-866e-4c742d50333b-kube-api-access-nbjrg\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.798844 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.798816 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-dshm\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.900770 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.900726 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778e00fa-3daa-47b9-866e-4c742d50333b-tls-certs\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.900985 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.900781 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-tmp-dir\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.900985 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.900802 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-home\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.900985 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.900822 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-model-cache\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.900985 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.900837 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjrg\" (UniqueName: \"kubernetes.io/projected/778e00fa-3daa-47b9-866e-4c742d50333b-kube-api-access-nbjrg\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.901223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.900991 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-dshm\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.901283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.901238 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-tmp-dir\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.901283 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.901254 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-model-cache\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.901362 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.901302 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-home\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.903290 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.903269 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-dshm\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.903726 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.903705 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778e00fa-3daa-47b9-866e-4c742d50333b-tls-certs\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:00.965233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:00.965154 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjrg\" (UniqueName: \"kubernetes.io/projected/778e00fa-3daa-47b9-866e-4c742d50333b-kube-api-access-nbjrg\") pod \"auth-test-model-4743f8-kserve-56f459d485-866ws\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:01.022181 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:01.022144 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:01.405849 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:01.405816 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws"] Oct 31 00:56:01.406122 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:56:01.406092 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod778e00fa_3daa_47b9_866e_4c742d50333b.slice/crio-c285fc977fdb4f5d658ff91cc440d169ada1ed13b1fbb7d5ff09a9be8b185a0f WatchSource:0}: Error finding container c285fc977fdb4f5d658ff91cc440d169ada1ed13b1fbb7d5ff09a9be8b185a0f: Status 404 returned error can't find the container with id c285fc977fdb4f5d658ff91cc440d169ada1ed13b1fbb7d5ff09a9be8b185a0f Oct 31 00:56:02.258494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:02.258460 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" event={"ID":"778e00fa-3daa-47b9-866e-4c742d50333b","Type":"ContainerStarted","Data":"08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6"} Oct 31 00:56:02.258494 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:02.258501 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" event={"ID":"778e00fa-3daa-47b9-866e-4c742d50333b","Type":"ContainerStarted","Data":"c285fc977fdb4f5d658ff91cc440d169ada1ed13b1fbb7d5ff09a9be8b185a0f"} Oct 31 00:56:02.381231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:02.381178 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" podStartSLOduration=2.381164584 podStartE2EDuration="2.381164584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:02.380740197 +0000 UTC m=+3194.856098632" watchObservedRunningTime="2026-10-31 00:56:02.381164584 +0000 UTC m=+3194.856523057" Oct 31 00:56:02.728527 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:56:02.728489 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:56:03.022763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:03.022681 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:04.081989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:04.081948 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:56:04.082383 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:04.082007 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:04.247369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:04.247333 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:56:04.247547 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:04.247400 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:04.672185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:04.672146 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:56:04.672358 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:04.672213 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:05.110348 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.110319 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:56:05.148816 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.148786 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:56:05.204346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.204321 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:56:05.211734 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.211708 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:56:05.255075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255049 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d31630-12eb-414a-8466-64283a778ff6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d31630-12eb-414a-8466-64283a778ff6-istiod-ca-cert\") pod \"44d31630-12eb-414a-8466-64283a778ff6\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " Oct 31 00:56:05.255263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255120 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-data\") pod \"742f0e21-1b98-4190-a352-262d1733434a\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " Oct 31 00:56:05.255263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255151 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44d31630-12eb-414a-8466-64283a778ff6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d31630-12eb-414a-8466-64283a778ff6-istio-podinfo\") pod \"44d31630-12eb-414a-8466-64283a778ff6\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " Oct 31 00:56:05.255263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255177 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/742f0e21-1b98-4190-a352-262d1733434a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742f0e21-1b98-4190-a352-262d1733434a-istiod-ca-cert\") pod \"742f0e21-1b98-4190-a352-262d1733434a\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " Oct 31 00:56:05.255263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255203 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-kube-api-access-v5zcl\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-kube-api-access-v5zcl\") pod \"44d31630-12eb-414a-8466-64283a778ff6\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " Oct 31 00:56:05.255263 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255245 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-istio-token\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-istio-token\") pod \"742f0e21-1b98-4190-a352-262d1733434a\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " Oct 31 00:56:05.255563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255290 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-socket\") pod \"44d31630-12eb-414a-8466-64283a778ff6\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " Oct 31 00:56:05.255563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255373 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-data\") pod \"44d31630-12eb-414a-8466-64283a778ff6\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " Oct 31 00:56:05.255563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255406 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-istio-token\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-istio-token\") pod \"44d31630-12eb-414a-8466-64283a778ff6\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " Oct 31 00:56:05.255563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255433 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-certs\") pod \"742f0e21-1b98-4190-a352-262d1733434a\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " Oct 31 00:56:05.255563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255467 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-credential-socket\") pod \"44d31630-12eb-414a-8466-64283a778ff6\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " Oct 31 00:56:05.255563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255494 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d31630-12eb-414a-8466-64283a778ff6-istiod-ca-cert" pod "44d31630-12eb-414a-8466-64283a778ff6" (UID: "44d31630-12eb-414a-8466-64283a778ff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:05.255973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255648 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-envoy\") pod \"742f0e21-1b98-4190-a352-262d1733434a\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " Oct 31 00:56:05.255973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255675 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742f0e21-1b98-4190-a352-262d1733434a-istiod-ca-cert" pod "742f0e21-1b98-4190-a352-262d1733434a" (UID: "742f0e21-1b98-4190-a352-262d1733434a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:05.255973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255707 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-kube-api-access-7xbrs\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-kube-api-access-7xbrs\") pod \"742f0e21-1b98-4190-a352-262d1733434a\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " Oct 31 00:56:05.255973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255814 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-credential-socket\") pod \"742f0e21-1b98-4190-a352-262d1733434a\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " Oct 31 00:56:05.255973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255868 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-certs\") pod \"44d31630-12eb-414a-8466-64283a778ff6\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " Oct 31 00:56:05.255973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255919 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/742f0e21-1b98-4190-a352-262d1733434a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742f0e21-1b98-4190-a352-262d1733434a-istio-podinfo\") pod \"742f0e21-1b98-4190-a352-262d1733434a\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " Oct 31 00:56:05.255973 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255954 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-socket\") pod \"742f0e21-1b98-4190-a352-262d1733434a\" (UID: \"742f0e21-1b98-4190-a352-262d1733434a\") " Oct 31 00:56:05.256312 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.255997 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-envoy\") pod \"44d31630-12eb-414a-8466-64283a778ff6\" (UID: \"44d31630-12eb-414a-8466-64283a778ff6\") " Oct 31 00:56:05.256372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.256357 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-data" pod "44d31630-12eb-414a-8466-64283a778ff6" (UID: "44d31630-12eb-414a-8466-64283a778ff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.256493 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.256467 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-credential-socket" pod "44d31630-12eb-414a-8466-64283a778ff6" (UID: "44d31630-12eb-414a-8466-64283a778ff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.256647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.256512 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-data" pod "742f0e21-1b98-4190-a352-262d1733434a" (UID: "742f0e21-1b98-4190-a352-262d1733434a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.256647 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.256625 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-certs" pod "742f0e21-1b98-4190-a352-262d1733434a" (UID: "742f0e21-1b98-4190-a352-262d1733434a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.256787 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.256704 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-certs" pod "44d31630-12eb-414a-8466-64283a778ff6" (UID: "44d31630-12eb-414a-8466-64283a778ff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.256882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.256860 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-credential-socket" pod "742f0e21-1b98-4190-a352-262d1733434a" (UID: "742f0e21-1b98-4190-a352-262d1733434a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.257086 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257064 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-socket" pod "742f0e21-1b98-4190-a352-262d1733434a" (UID: "742f0e21-1b98-4190-a352-262d1733434a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.257178 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257127 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-socket" pod "44d31630-12eb-414a-8466-64283a778ff6" (UID: "44d31630-12eb-414a-8466-64283a778ff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.257345 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257321 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.257426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257346 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d31630-12eb-414a-8466-64283a778ff6-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.257426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257362 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.257426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257379 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742f0e21-1b98-4190-a352-262d1733434a-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.257426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257393 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.257426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257406 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.257426 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257420 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.257767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257436 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.257767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257452 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.257767 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.257466 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.259198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.259104 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-envoy" pod "44d31630-12eb-414a-8466-64283a778ff6" (UID: "44d31630-12eb-414a-8466-64283a778ff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.259198 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.259143 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44d31630-12eb-414a-8466-64283a778ff6-istio-podinfo" pod "44d31630-12eb-414a-8466-64283a778ff6" (UID: "44d31630-12eb-414a-8466-64283a778ff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:05.259364 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.259197 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-kube-api-access-v5zcl" pod "44d31630-12eb-414a-8466-64283a778ff6" (UID: "44d31630-12eb-414a-8466-64283a778ff6"). InnerVolumeSpecName "kube-api-access-v5zcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.259504 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.259476 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-istio-token" pod "742f0e21-1b98-4190-a352-262d1733434a" (UID: "742f0e21-1b98-4190-a352-262d1733434a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.259934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.259914 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-kube-api-access-7xbrs" pod "742f0e21-1b98-4190-a352-262d1733434a" (UID: "742f0e21-1b98-4190-a352-262d1733434a"). InnerVolumeSpecName "kube-api-access-7xbrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.260099 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.260079 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-istio-token" pod "44d31630-12eb-414a-8466-64283a778ff6" (UID: "44d31630-12eb-414a-8466-64283a778ff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.260719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.260702 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-envoy" pod "742f0e21-1b98-4190-a352-262d1733434a" (UID: "742f0e21-1b98-4190-a352-262d1733434a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.260829 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.260808 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/742f0e21-1b98-4190-a352-262d1733434a-istio-podinfo" pod "742f0e21-1b98-4190-a352-262d1733434a" (UID: "742f0e21-1b98-4190-a352-262d1733434a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:05.288034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.287990 4969 generic.go:356] "Generic (PLEG): container finished" podID="44d31630-12eb-414a-8466-64283a778ff6" containerID="65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a" exitCode=0 Oct 31 00:56:05.288207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.288043 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" Oct 31 00:56:05.288207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.288051 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" event={"ID":"44d31630-12eb-414a-8466-64283a778ff6","Type":"ContainerDied","Data":"65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a"} Oct 31 00:56:05.288207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.288102 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc" event={"ID":"44d31630-12eb-414a-8466-64283a778ff6","Type":"ContainerDied","Data":"f4bccbd4691c8e6b31d4c5dff68b83356585e04db6bedf65bcd09ca8f68fac53"} Oct 31 00:56:05.288207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.288116 4969 scope.go:122] "RemoveContainer" containerID="65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a" Oct 31 00:56:05.294453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.293929 4969 generic.go:356] "Generic (PLEG): container finished" podID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerID="11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e" exitCode=0 Oct 31 00:56:05.294453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.294006 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" event={"ID":"6d824ac7-d126-407d-b68f-10869bd7a0f4","Type":"ContainerDied","Data":"11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e"} Oct 31 00:56:05.294453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.294039 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" event={"ID":"6d824ac7-d126-407d-b68f-10869bd7a0f4","Type":"ContainerDied","Data":"b433997365f92e56fcab774481f873aa7aed2caa3975113b1b2b66478cd45e9f"} Oct 31 00:56:05.294453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.294120 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv" Oct 31 00:56:05.295962 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.295937 4969 generic.go:356] "Generic (PLEG): container finished" podID="742f0e21-1b98-4190-a352-262d1733434a" containerID="3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52" exitCode=0 Oct 31 00:56:05.296096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.295973 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" event={"ID":"742f0e21-1b98-4190-a352-262d1733434a","Type":"ContainerDied","Data":"3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52"} Oct 31 00:56:05.296096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.295992 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" Oct 31 00:56:05.296096 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.296032 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" event={"ID":"742f0e21-1b98-4190-a352-262d1733434a","Type":"ContainerDied","Data":"a2472bea8ea8bc4b4a43929542deefe7e70dadf5fa7e7684569c2df5705338f2"} Oct 31 00:56:05.297772 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.297743 4969 generic.go:356] "Generic (PLEG): container finished" podID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerID="6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4" exitCode=0 Oct 31 00:56:05.297904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.297786 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" event={"ID":"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54","Type":"ContainerDied","Data":"6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4"} Oct 31 00:56:05.297904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.297813 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" event={"ID":"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54","Type":"ContainerDied","Data":"677c83b4b7d7fe1aa2bfec988f399c6d4a2a78532d2d55be42784c7251810c98"} Oct 31 00:56:05.297904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.297857 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6" Oct 31 00:56:05.307363 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.307346 4969 scope.go:122] "RemoveContainer" containerID="65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a" Oct 31 00:56:05.307691 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:56:05.307670 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a\": container with ID starting with 65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a not found: ID does not exist" containerID="65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a" Oct 31 00:56:05.307799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.307700 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a"} err="failed to get container status \"65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a\": rpc error: code = NotFound desc = could not find container \"65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a\": container with ID starting with 65f14686592978b51ae7cd287b163ae46e693c271376993321b18643f905c78a not found: ID does not exist" Oct 31 00:56:05.307799 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.307722 4969 scope.go:122] "RemoveContainer" containerID="11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e" Oct 31 00:56:05.317950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.317931 4969 scope.go:122] "RemoveContainer" containerID="11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e" Oct 31 00:56:05.318238 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:56:05.318219 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e\": container with ID starting with 11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e not found: ID does not exist" containerID="11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e" Oct 31 00:56:05.318331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.318245 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e"} err="failed to get container status \"11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e\": rpc error: code = NotFound desc = could not find container \"11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e\": container with ID starting with 11a080234c66765f40ab9789bcef244043cf219fb54b78d6fe01d0e3659c6c6e not found: ID does not exist" Oct 31 00:56:05.318331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.318262 4969 scope.go:122] "RemoveContainer" containerID="3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52" Oct 31 00:56:05.330146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.330128 4969 scope.go:122] "RemoveContainer" containerID="3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52" Oct 31 00:56:05.330404 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:56:05.330388 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52\": container with ID starting with 3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52 not found: ID does not exist" containerID="3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52" Oct 31 00:56:05.330463 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.330415 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52"} err="failed to get container status \"3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52\": rpc error: code = NotFound desc = could not find container \"3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52\": container with ID starting with 3154000153b54e83b452e6fe31f6100216ad5c9a0a9fb02c2acb43df64d31a52 not found: ID does not exist" Oct 31 00:56:05.330463 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.330434 4969 scope.go:122] "RemoveContainer" containerID="6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4" Oct 31 00:56:05.340034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.340009 4969 scope.go:122] "RemoveContainer" containerID="6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4" Oct 31 00:56:05.340297 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:56:05.340279 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4\": container with ID starting with 6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4 not found: ID does not exist" containerID="6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4" Oct 31 00:56:05.340361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.340304 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4"} err="failed to get container status \"6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4\": rpc error: code = NotFound desc = could not find container \"6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4\": container with ID starting with 6305aa02f4dd5483a73d6eed0d48b823e56e9ad1594087f6c09c04d15926e1f4 not found: ID does not exist" Oct 31 00:56:05.358780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.358758 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-socket\") pod \"6d824ac7-d126-407d-b68f-10869bd7a0f4\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " Oct 31 00:56:05.358875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.358797 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-socket\") pod \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " Oct 31 00:56:05.358875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.358834 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-kube-api-access-b6fcx\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-kube-api-access-b6fcx\") pod \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " Oct 31 00:56:05.358991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.358873 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istiod-ca-cert\") pod \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " Oct 31 00:56:05.358991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.358894 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-envoy\") pod \"6d824ac7-d126-407d-b68f-10869bd7a0f4\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " Oct 31 00:56:05.358991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.358915 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-certs\") pod \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " Oct 31 00:56:05.358991 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.358960 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-data\") pod \"6d824ac7-d126-407d-b68f-10869bd7a0f4\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " Oct 31 00:56:05.359209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.358992 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-podinfo\") pod \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " Oct 31 00:56:05.359209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359002 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-socket" pod "6d824ac7-d126-407d-b68f-10869bd7a0f4" (UID: "6d824ac7-d126-407d-b68f-10869bd7a0f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.359209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359022 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-certs\") pod \"6d824ac7-d126-407d-b68f-10869bd7a0f4\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " Oct 31 00:56:05.359209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359089 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-kube-api-access-dqnmg\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-kube-api-access-dqnmg\") pod \"6d824ac7-d126-407d-b68f-10869bd7a0f4\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " Oct 31 00:56:05.359209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359115 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-socket" pod "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" (UID: "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.359209 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359191 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-credential-socket\") pod \"6d824ac7-d126-407d-b68f-10869bd7a0f4\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " Oct 31 00:56:05.359528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359219 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-credential-socket\") pod \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " Oct 31 00:56:05.359528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359259 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-podinfo\") pod \"6d824ac7-d126-407d-b68f-10869bd7a0f4\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " Oct 31 00:56:05.359528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359289 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d824ac7-d126-407d-b68f-10869bd7a0f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d824ac7-d126-407d-b68f-10869bd7a0f4-istiod-ca-cert\") pod \"6d824ac7-d126-407d-b68f-10869bd7a0f4\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " Oct 31 00:56:05.359528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359321 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-certs" pod "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" (UID: "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.359528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359353 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-envoy\") pod \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " Oct 31 00:56:05.359528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359381 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-data\") pod \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " Oct 31 00:56:05.359528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359442 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-token\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-token\") pod \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\" (UID: \"c81ec4b1-24ea-4bb9-bf45-64badd7bfc54\") " Oct 31 00:56:05.359528 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359480 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-token\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-token\") pod \"6d824ac7-d126-407d-b68f-10869bd7a0f4\" (UID: \"6d824ac7-d126-407d-b68f-10869bd7a0f4\") " Oct 31 00:56:05.360001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359352 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istiod-ca-cert" pod "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" (UID: "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:05.360001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359518 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-credential-socket" pod "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" (UID: "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.360001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359648 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-data" pod "6d824ac7-d126-407d-b68f-10869bd7a0f4" (UID: "6d824ac7-d126-407d-b68f-10869bd7a0f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.360001 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.359792 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-credential-socket" pod "6d824ac7-d126-407d-b68f-10869bd7a0f4" (UID: "6d824ac7-d126-407d-b68f-10869bd7a0f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360069 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d31630-12eb-414a-8466-64283a778ff6-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360093 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360109 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360123 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5zcl\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-kube-api-access-v5zcl\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360139 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360152 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d31630-12eb-414a-8466-64283a778ff6-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360152 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-certs" pod "6d824ac7-d126-407d-b68f-10869bd7a0f4" (UID: "6d824ac7-d126-407d-b68f-10869bd7a0f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360105 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d824ac7-d126-407d-b68f-10869bd7a0f4-istiod-ca-cert" pod "6d824ac7-d126-407d-b68f-10869bd7a0f4" (UID: "6d824ac7-d126-407d-b68f-10869bd7a0f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360167 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742f0e21-1b98-4190-a352-262d1733434a-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360224 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbrs\" (UniqueName: \"kubernetes.io/projected/742f0e21-1b98-4190-a352-262d1733434a-kube-api-access-7xbrs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360243 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360260 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360274 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360291 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360308 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742f0e21-1b98-4190-a352-262d1733434a-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360323 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.360400 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360340 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d31630-12eb-414a-8466-64283a778ff6-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.361404 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.360618 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-data" pod "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" (UID: "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.361931 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.361906 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-kube-api-access-b6fcx" pod "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" (UID: "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54"). InnerVolumeSpecName "kube-api-access-b6fcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.362259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.362234 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-podinfo" pod "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" (UID: "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:05.362361 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.362308 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-envoy" pod "6d824ac7-d126-407d-b68f-10869bd7a0f4" (UID: "6d824ac7-d126-407d-b68f-10869bd7a0f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.362415 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.362361 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-podinfo" pod "6d824ac7-d126-407d-b68f-10869bd7a0f4" (UID: "6d824ac7-d126-407d-b68f-10869bd7a0f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:05.362927 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.362897 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-token" pod "6d824ac7-d126-407d-b68f-10869bd7a0f4" (UID: "6d824ac7-d126-407d-b68f-10869bd7a0f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.363014 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.362968 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-envoy" pod "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" (UID: "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.363223 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.363205 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-kube-api-access-dqnmg" pod "6d824ac7-d126-407d-b68f-10869bd7a0f4" (UID: "6d824ac7-d126-407d-b68f-10869bd7a0f4"). InnerVolumeSpecName "kube-api-access-dqnmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.363797 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.363779 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-token" pod "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" (UID: "c81ec4b1-24ea-4bb9-bf45-64badd7bfc54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.461703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461620 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.461703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461649 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.461703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461659 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d824ac7-d126-407d-b68f-10869bd7a0f4-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.461703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461668 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqnmg\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-kube-api-access-dqnmg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.461703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461677 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.461703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461687 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d824ac7-d126-407d-b68f-10869bd7a0f4-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.461703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461695 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.461703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461705 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.462079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461715 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.462079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461723 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d824ac7-d126-407d-b68f-10869bd7a0f4-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.462079 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.461733 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6fcx\" (UniqueName: \"kubernetes.io/projected/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54-kube-api-access-b6fcx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.754909 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.754814 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv"] Oct 31 00:56:05.759341 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.759314 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-6tthv"] Oct 31 00:56:05.868456 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.868419 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6"] Oct 31 00:56:05.881336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:05.876413 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-8l8m6"] Oct 31 00:56:06.009755 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:06.009661 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d"] Oct 31 00:56:06.013087 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:06.013063 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d"] Oct 31 00:56:06.100239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:06.100208 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc"] Oct 31 00:56:06.105858 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:06.105826 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-qt6lc"] Oct 31 00:56:06.129453 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:06.129410 4969 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:56:06.129854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:06.129485 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-87x5d" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:56:07.735349 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:07.735306 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d31630-12eb-414a-8466-64283a778ff6" path="/var/lib/kubelet/pods/44d31630-12eb-414a-8466-64283a778ff6/volumes" Oct 31 00:56:07.736050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:07.736027 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" path="/var/lib/kubelet/pods/6d824ac7-d126-407d-b68f-10869bd7a0f4/volumes" Oct 31 00:56:07.736531 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:07.736510 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="742f0e21-1b98-4190-a352-262d1733434a" path="/var/lib/kubelet/pods/742f0e21-1b98-4190-a352-262d1733434a/volumes" Oct 31 00:56:07.737356 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:07.737333 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" path="/var/lib/kubelet/pods/c81ec4b1-24ea-4bb9-bf45-64badd7bfc54/volumes" Oct 31 00:56:11.022477 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:11.022436 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:11.035183 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:11.035157 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:11.357623 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:11.357522 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 00:56:14.201138 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201104 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm"] Oct 31 00:56:14.201698 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201684 4969 cpu_manager.go:414] "containerMap: removing container" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" Oct 31 00:56:14.201766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201699 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" Oct 31 00:56:14.201766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201719 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" Oct 31 00:56:14.201766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201724 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c58d5b-e672-4e93-8bfe-4f7677c31125" containerName="istio-proxy" Oct 31 00:56:14.201766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201746 4969 cpu_manager.go:414] "containerMap: removing container" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" Oct 31 00:56:14.201766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201751 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" Oct 31 00:56:14.201766 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201764 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" Oct 31 00:56:14.201952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201769 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" Oct 31 00:56:14.201952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201783 4969 cpu_manager.go:414] "containerMap: removing container" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" Oct 31 00:56:14.201952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201787 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" Oct 31 00:56:14.201952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201873 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81ec4b1-24ea-4bb9-bf45-64badd7bfc54" containerName="istio-proxy" Oct 31 00:56:14.201952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201884 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d824ac7-d126-407d-b68f-10869bd7a0f4" containerName="istio-proxy" Oct 31 00:56:14.201952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201891 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="742f0e21-1b98-4190-a352-262d1733434a" containerName="istio-proxy" Oct 31 00:56:14.201952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.201900 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d31630-12eb-414a-8466-64283a778ff6" containerName="istio-proxy" Oct 31 00:56:14.207074 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.207054 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.264063 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.264029 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-home\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.264239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.264081 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf64996-5a62-4fde-9dcf-048c85345798-tls-certs\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.264239 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.264156 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-model-cache\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.264372 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.264343 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw66s\" (UniqueName: \"kubernetes.io/projected/edf64996-5a62-4fde-9dcf-048c85345798-kube-api-access-qw66s\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.264498 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.264476 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-dshm\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.264537 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.264518 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-tmp-dir\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.309016 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.308983 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ee480f\"/\"kube-root-ca.crt\"" Oct 31 00:56:14.309573 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.309556 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ee480f\"/\"openshift-service-ca.crt\"" Oct 31 00:56:14.309735 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.309628 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ee480f\"/\"default-dockercfg-vrxmc\"" Oct 31 00:56:14.309933 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.309918 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ee480f\"/\"noten-model-ee480f-kserve-self-signed-certs\"" Oct 31 00:56:14.310992 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.310971 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm"] Oct 31 00:56:14.365913 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.365878 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-home\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.366127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.365935 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf64996-5a62-4fde-9dcf-048c85345798-tls-certs\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.366127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.365982 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-model-cache\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.366127 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.366076 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw66s\" (UniqueName: \"kubernetes.io/projected/edf64996-5a62-4fde-9dcf-048c85345798-kube-api-access-qw66s\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.366316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.366176 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-dshm\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.366316 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.366200 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-tmp-dir\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.366422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.366327 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-home\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.366422 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.366342 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-model-cache\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.366579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.366561 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-tmp-dir\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.368563 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.368534 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-dshm\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.368954 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.368935 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf64996-5a62-4fde-9dcf-048c85345798-tls-certs\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.427010 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.426970 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw66s\" (UniqueName: \"kubernetes.io/projected/edf64996-5a62-4fde-9dcf-048c85345798-kube-api-access-qw66s\") pod \"noten-model-ee480f-kserve-7c8f99c7cc-dnsbm\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.519904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.519819 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:14.904306 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:14.904274 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm"] Oct 31 00:56:14.905222 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:56:14.905196 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedf64996_5a62_4fde_9dcf_048c85345798.slice/crio-2c3302abab110cf06a5713438f1acd813662c9793712dbb42a081e03ca9dd981 WatchSource:0}: Error finding container 2c3302abab110cf06a5713438f1acd813662c9793712dbb42a081e03ca9dd981: Status 404 returned error can't find the container with id 2c3302abab110cf06a5713438f1acd813662c9793712dbb42a081e03ca9dd981 Oct 31 00:56:15.381434 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:15.381395 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" event={"ID":"edf64996-5a62-4fde-9dcf-048c85345798","Type":"ContainerStarted","Data":"f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79"} Oct 31 00:56:15.381839 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:15.381444 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" event={"ID":"edf64996-5a62-4fde-9dcf-048c85345798","Type":"ContainerStarted","Data":"2c3302abab110cf06a5713438f1acd813662c9793712dbb42a081e03ca9dd981"} Oct 31 00:56:15.500174 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:15.500112 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" podStartSLOduration=1.5000954100000001 podStartE2EDuration="1.50009541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:15.498972557 +0000 UTC m=+3207.974330981" watchObservedRunningTime="2026-10-31 00:56:15.50009541 +0000 UTC m=+3207.975453836" Oct 31 00:56:15.520333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:15.520309 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:24.520356 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:24.520323 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:24.533375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:24.533349 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:25.478545 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:25.478516 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:56:32.727765 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:56:32.727727 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:56:44.708959 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.708929 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w"] Oct 31 00:56:44.713444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.713422 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.769204 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.769132 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e1a6c\"/\"kube-root-ca.crt\"" Oct 31 00:56:44.818451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.818418 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e1a6c\"/\"default-dockercfg-8q28v\"" Oct 31 00:56:44.818717 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.818694 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e1a6c\"/\"auth-test-model-4e1a6c-kserve-self-signed-certs\"" Oct 31 00:56:44.819034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.819012 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e1a6c\"/\"openshift-service-ca.crt\"" Oct 31 00:56:44.819707 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.819683 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-model-cache\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.819813 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.819744 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tmp-dir\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.819882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.819831 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-home\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.819882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.819862 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-dshm\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.820011 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.819987 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tls-certs\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.820100 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.820027 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77vp7\" (UniqueName: \"kubernetes.io/projected/4796847a-ea8b-4fc8-8c21-8b7d145866f8-kube-api-access-77vp7\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.822509 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.822488 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w"] Oct 31 00:56:44.920874 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.920836 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tls-certs\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.920874 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.920882 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77vp7\" (UniqueName: \"kubernetes.io/projected/4796847a-ea8b-4fc8-8c21-8b7d145866f8-kube-api-access-77vp7\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.921146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.920936 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-model-cache\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.921146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.920974 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tmp-dir\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.921146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.921024 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-home\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.921146 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.921062 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-dshm\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.921444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.921421 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tmp-dir\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.921444 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.921436 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-home\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.921561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.921490 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-model-cache\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.923507 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.923482 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-dshm\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.923814 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.923794 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tls-certs\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:44.982895 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:44.982869 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77vp7\" (UniqueName: \"kubernetes.io/projected/4796847a-ea8b-4fc8-8c21-8b7d145866f8-kube-api-access-77vp7\") pod \"auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:45.027346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:45.027280 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:45.418221 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:56:45.418187 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4796847a_ea8b_4fc8_8c21_8b7d145866f8.slice/crio-d83c8d15b82bc841e173172d67661d2872e615afa28218ffd2b78fcfca71ffbf WatchSource:0}: Error finding container d83c8d15b82bc841e173172d67661d2872e615afa28218ffd2b78fcfca71ffbf: Status 404 returned error can't find the container with id d83c8d15b82bc841e173172d67661d2872e615afa28218ffd2b78fcfca71ffbf Oct 31 00:56:45.493225 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:45.493197 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w"] Oct 31 00:56:45.612694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:45.612656 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" event={"ID":"4796847a-ea8b-4fc8-8c21-8b7d145866f8","Type":"ContainerStarted","Data":"d4710bbe6947be3f759b68c17708c04d506036b53dd4d5cbd04ec9a969b2cf86"} Oct 31 00:56:45.612694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:45.612697 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" event={"ID":"4796847a-ea8b-4fc8-8c21-8b7d145866f8","Type":"ContainerStarted","Data":"d83c8d15b82bc841e173172d67661d2872e615afa28218ffd2b78fcfca71ffbf"} Oct 31 00:56:45.784310 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:45.784216 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" podStartSLOduration=1.784203721 podStartE2EDuration="1.784203721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:45.733458718 +0000 UTC m=+3238.208817141" watchObservedRunningTime="2026-10-31 00:56:45.784203721 +0000 UTC m=+3238.259562160" Oct 31 00:56:46.027432 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:46.027404 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:49.843533 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:49.843502 4969 scope.go:122] "RemoveContainer" containerID="8b69cf6817f27b5eaf85102c9ffbc8a216bad592459d107df53f64ee59cfad6c" Oct 31 00:56:49.854235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:49.854211 4969 scope.go:122] "RemoveContainer" containerID="83ade6325d45670cc6bd2ba9db969fa30275288600c5954fa0ab02610ef06cb9" Oct 31 00:56:54.770578 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:54.770544 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd"] Oct 31 00:56:54.771047 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:54.770777 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" podUID="91c35be1-e5b3-4873-871d-275f05ce0dae" containerName="istio-proxy" containerID="cri-o://ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535" gracePeriod=30 Oct 31 00:56:55.027653 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:55.027544 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:55.040398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:55.040372 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:56:55.698179 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:56:55.698148 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 00:57:00.037270 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.037211 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:57:00.222742 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.222711 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-token\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-token\") pod \"91c35be1-e5b3-4873-871d-275f05ce0dae\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " Oct 31 00:57:00.222953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.222762 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-socket\") pod \"91c35be1-e5b3-4873-871d-275f05ce0dae\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " Oct 31 00:57:00.222953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.222790 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-podinfo\") pod \"91c35be1-e5b3-4873-871d-275f05ce0dae\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " Oct 31 00:57:00.222953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.222868 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-certs\") pod \"91c35be1-e5b3-4873-871d-275f05ce0dae\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " Oct 31 00:57:00.222953 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.222941 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-data\") pod \"91c35be1-e5b3-4873-871d-275f05ce0dae\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " Oct 31 00:57:00.223271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223037 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-kube-api-access-l6vhn\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-kube-api-access-l6vhn\") pod \"91c35be1-e5b3-4873-871d-275f05ce0dae\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " Oct 31 00:57:00.223271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223111 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-credential-socket\") pod \"91c35be1-e5b3-4873-871d-275f05ce0dae\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " Oct 31 00:57:00.223271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223097 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-socket" pod "91c35be1-e5b3-4873-871d-275f05ce0dae" (UID: "91c35be1-e5b3-4873-871d-275f05ce0dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.223271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223141 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c35be1-e5b3-4873-871d-275f05ce0dae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c35be1-e5b3-4873-871d-275f05ce0dae-istiod-ca-cert\") pod \"91c35be1-e5b3-4873-871d-275f05ce0dae\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " Oct 31 00:57:00.223271 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223195 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-envoy\") pod \"91c35be1-e5b3-4873-871d-275f05ce0dae\" (UID: \"91c35be1-e5b3-4873-871d-275f05ce0dae\") " Oct 31 00:57:00.223535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223397 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-credential-socket" pod "91c35be1-e5b3-4873-871d-275f05ce0dae" (UID: "91c35be1-e5b3-4873-871d-275f05ce0dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.223535 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223453 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c35be1-e5b3-4873-871d-275f05ce0dae-istiod-ca-cert" pod "91c35be1-e5b3-4873-871d-275f05ce0dae" (UID: "91c35be1-e5b3-4873-871d-275f05ce0dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:00.223716 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223681 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-data" pod "91c35be1-e5b3-4873-871d-275f05ce0dae" (UID: "91c35be1-e5b3-4873-871d-275f05ce0dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.223921 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223899 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.223921 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223922 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.224125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223937 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c35be1-e5b3-4873-871d-275f05ce0dae-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.224125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.223949 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.224125 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.224013 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-certs" pod "91c35be1-e5b3-4873-871d-275f05ce0dae" (UID: "91c35be1-e5b3-4873-871d-275f05ce0dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.225784 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.225756 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-envoy" pod "91c35be1-e5b3-4873-871d-275f05ce0dae" (UID: "91c35be1-e5b3-4873-871d-275f05ce0dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:00.225950 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.225926 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-token" pod "91c35be1-e5b3-4873-871d-275f05ce0dae" (UID: "91c35be1-e5b3-4873-871d-275f05ce0dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:00.226037 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.225930 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-podinfo" pod "91c35be1-e5b3-4873-871d-275f05ce0dae" (UID: "91c35be1-e5b3-4873-871d-275f05ce0dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:57:00.226207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.226187 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-kube-api-access-l6vhn" pod "91c35be1-e5b3-4873-871d-275f05ce0dae" (UID: "91c35be1-e5b3-4873-871d-275f05ce0dae"). InnerVolumeSpecName "kube-api-access-l6vhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:00.274805 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.274780 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm"] Oct 31 00:57:00.325455 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.325389 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.325455 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.325416 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.325455 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.325426 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.325455 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.325435 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6vhn\" (UniqueName: \"kubernetes.io/projected/91c35be1-e5b3-4873-871d-275f05ce0dae-kube-api-access-l6vhn\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.325455 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.325444 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c35be1-e5b3-4873-871d-275f05ce0dae-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:00.724886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.724852 4969 generic.go:356] "Generic (PLEG): container finished" podID="91c35be1-e5b3-4873-871d-275f05ce0dae" containerID="ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535" exitCode=0 Oct 31 00:57:00.725075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.724910 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" Oct 31 00:57:00.725075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.724943 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" event={"ID":"91c35be1-e5b3-4873-871d-275f05ce0dae","Type":"ContainerDied","Data":"ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535"} Oct 31 00:57:00.725075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.725008 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd" event={"ID":"91c35be1-e5b3-4873-871d-275f05ce0dae","Type":"ContainerDied","Data":"284fb82ee5f7d52ba5e46e92dca0589445c554f1b58e17cbb4a15e98f2d520e6"} Oct 31 00:57:00.725075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.725030 4969 scope.go:122] "RemoveContainer" containerID="ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535" Oct 31 00:57:00.736622 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.736586 4969 scope.go:122] "RemoveContainer" containerID="ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535" Oct 31 00:57:00.740756 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:00.740722 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535\": container with ID starting with ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535 not found: ID does not exist" containerID="ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535" Oct 31 00:57:00.740884 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.740764 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535"} err="failed to get container status \"ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535\": rpc error: code = NotFound desc = could not find container \"ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535\": container with ID starting with ca25bbf3765f0a7526142989002e5215ff165b1508423fe1f69a8f1c57e3b535 not found: ID does not exist" Oct 31 00:57:00.893722 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.893690 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd"] Oct 31 00:57:00.947699 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:00.947671 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ee480f-openshift-default-fbf7769b9-9h2xd"] Oct 31 00:57:01.742351 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:01.742302 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c35be1-e5b3-4873-871d-275f05ce0dae" path="/var/lib/kubelet/pods/91c35be1-e5b3-4873-871d-275f05ce0dae/volumes" Oct 31 00:57:02.729626 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:02.729563 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:57:15.296210 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.296171 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" podUID="edf64996-5a62-4fde-9dcf-048c85345798" containerName="main" containerID="cri-o://f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79" gracePeriod=45 Oct 31 00:57:15.467464 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.467423 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" podUID="edf64996-5a62-4fde-9dcf-048c85345798" containerName="main" probeResult="failure" output="Get \"https://10.133.0.230:8000/ready\": dial tcp 10.133.0.230:8000: connect: connection refused" Oct 31 00:57:15.566561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.566498 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:57:15.700397 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.700363 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-model-cache\") pod \"edf64996-5a62-4fde-9dcf-048c85345798\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " Oct 31 00:57:15.700614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.700481 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-tmp-dir\") pod \"edf64996-5a62-4fde-9dcf-048c85345798\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " Oct 31 00:57:15.700614 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.700580 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-dshm\") pod \"edf64996-5a62-4fde-9dcf-048c85345798\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " Oct 31 00:57:15.700756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.700650 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf64996-5a62-4fde-9dcf-048c85345798-tls-certs\" (UniqueName: \"kubernetes.io/secret/edf64996-5a62-4fde-9dcf-048c85345798-tls-certs\") pod \"edf64996-5a62-4fde-9dcf-048c85345798\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " Oct 31 00:57:15.700756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.700709 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf64996-5a62-4fde-9dcf-048c85345798-kube-api-access-qw66s\" (UniqueName: \"kubernetes.io/projected/edf64996-5a62-4fde-9dcf-048c85345798-kube-api-access-qw66s\") pod \"edf64996-5a62-4fde-9dcf-048c85345798\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " Oct 31 00:57:15.700756 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.700723 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-model-cache" pod "edf64996-5a62-4fde-9dcf-048c85345798" (UID: "edf64996-5a62-4fde-9dcf-048c85345798"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:15.700941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.700766 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-home\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-home\") pod \"edf64996-5a62-4fde-9dcf-048c85345798\" (UID: \"edf64996-5a62-4fde-9dcf-048c85345798\") " Oct 31 00:57:15.700941 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.700783 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-tmp-dir" pod "edf64996-5a62-4fde-9dcf-048c85345798" (UID: "edf64996-5a62-4fde-9dcf-048c85345798"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:15.701280 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.701254 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-home" pod "edf64996-5a62-4fde-9dcf-048c85345798" (UID: "edf64996-5a62-4fde-9dcf-048c85345798"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:15.701525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.701451 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:15.701525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.701475 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:15.701525 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.701491 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:15.705715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.705691 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf64996-5a62-4fde-9dcf-048c85345798-tls-certs" pod "edf64996-5a62-4fde-9dcf-048c85345798" (UID: "edf64996-5a62-4fde-9dcf-048c85345798"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:15.705715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.705700 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-dshm" pod "edf64996-5a62-4fde-9dcf-048c85345798" (UID: "edf64996-5a62-4fde-9dcf-048c85345798"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:15.705877 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.705816 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf64996-5a62-4fde-9dcf-048c85345798-kube-api-access-qw66s" pod "edf64996-5a62-4fde-9dcf-048c85345798" (UID: "edf64996-5a62-4fde-9dcf-048c85345798"). InnerVolumeSpecName "kube-api-access-qw66s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:15.802968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.802890 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf64996-5a62-4fde-9dcf-048c85345798-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:15.802968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.802924 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf64996-5a62-4fde-9dcf-048c85345798-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:15.802968 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.802940 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw66s\" (UniqueName: \"kubernetes.io/projected/edf64996-5a62-4fde-9dcf-048c85345798-kube-api-access-qw66s\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:15.852695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.852585 4969 generic.go:356] "Generic (PLEG): container finished" podID="edf64996-5a62-4fde-9dcf-048c85345798" containerID="f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79" exitCode=0 Oct 31 00:57:15.852695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.852631 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" event={"ID":"edf64996-5a62-4fde-9dcf-048c85345798","Type":"ContainerDied","Data":"f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79"} Oct 31 00:57:15.852695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.852667 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" Oct 31 00:57:15.852695 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.852682 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm" event={"ID":"edf64996-5a62-4fde-9dcf-048c85345798","Type":"ContainerDied","Data":"2c3302abab110cf06a5713438f1acd813662c9793712dbb42a081e03ca9dd981"} Oct 31 00:57:15.853005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.852700 4969 scope.go:122] "RemoveContainer" containerID="f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79" Oct 31 00:57:15.864492 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.864473 4969 scope.go:122] "RemoveContainer" containerID="f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79" Oct 31 00:57:15.864859 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:15.864827 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79\": container with ID starting with f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79 not found: ID does not exist" containerID="f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79" Oct 31 00:57:15.864948 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:15.864868 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79"} err="failed to get container status \"f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79\": rpc error: code = NotFound desc = could not find container \"f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79\": container with ID starting with f75ff5c11f866c21f916f2d74c88cd7dd3a9349bbfe790c07ea04ba4fb732f79 not found: ID does not exist" Oct 31 00:57:16.020922 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:16.020885 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm"] Oct 31 00:57:16.067948 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:16.067916 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ee480f/noten-model-ee480f-kserve-7c8f99c7cc-dnsbm"] Oct 31 00:57:17.735075 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:17.735044 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf64996-5a62-4fde-9dcf-048c85345798" path="/var/lib/kubelet/pods/edf64996-5a62-4fde-9dcf-048c85345798/volumes" Oct 31 00:57:23.153544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.153510 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4"] Oct 31 00:57:23.154185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.154168 4969 cpu_manager.go:414] "containerMap: removing container" podUID="edf64996-5a62-4fde-9dcf-048c85345798" containerName="main" Oct 31 00:57:23.154185 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.154185 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf64996-5a62-4fde-9dcf-048c85345798" containerName="main" Oct 31 00:57:23.154309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.154197 4969 cpu_manager.go:414] "containerMap: removing container" podUID="91c35be1-e5b3-4873-871d-275f05ce0dae" containerName="istio-proxy" Oct 31 00:57:23.154309 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.154202 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c35be1-e5b3-4873-871d-275f05ce0dae" containerName="istio-proxy" Oct 31 00:57:23.154399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.154314 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf64996-5a62-4fde-9dcf-048c85345798" containerName="main" Oct 31 00:57:23.154399 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.154324 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c35be1-e5b3-4873-871d-275f05ce0dae" containerName="istio-proxy" Oct 31 00:57:23.157769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.157749 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.260882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.260839 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-bf325c\"/\"openshift-service-ca.crt\"" Oct 31 00:57:23.260882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.260856 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-bf325c\"/\"kube-root-ca.crt\"" Oct 31 00:57:23.261161 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.261144 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-bf325c\"/\"auth-test-model-bf325c-kserve-self-signed-certs\"" Oct 31 00:57:23.261226 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.261189 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-bf325c\"/\"default-dockercfg-9dqc8\"" Oct 31 00:57:23.264916 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.264888 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4"] Oct 31 00:57:23.293678 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.293641 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-dshm\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.295027 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.294995 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tls-certs\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.295231 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.295211 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmfsn\" (UniqueName: \"kubernetes.io/projected/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-kube-api-access-fmfsn\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.296436 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.296405 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-home\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.296544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.296498 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tmp-dir\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.296643 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.296567 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-model-cache\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.397890 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.397855 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-model-cache\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.398073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.397913 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-dshm\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.398073 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.398020 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tls-certs\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.398162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.398072 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmfsn\" (UniqueName: \"kubernetes.io/projected/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-kube-api-access-fmfsn\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.398162 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.398148 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-home\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.398265 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.398184 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tmp-dir\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.398331 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.398289 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-model-cache\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.398497 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.398475 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-home\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.398579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.398556 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tmp-dir\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.400579 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.400556 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-dshm\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.400729 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.400711 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tls-certs\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.459311 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.459238 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmfsn\" (UniqueName: \"kubernetes.io/projected/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-kube-api-access-fmfsn\") pod \"auth-test-model-bf325c-kserve-758bb96ccc-bg7j4\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.470945 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.470921 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:23.852935 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.852906 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4"] Oct 31 00:57:23.854286 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:57:23.854258 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60ad44f3_a7fe_4664_8ae5_538e73aec5fe.slice/crio-bc75102cce59034186d7a2e87e30faf6fbb0bf3eda8a83a780cf19dfe809b850 WatchSource:0}: Error finding container bc75102cce59034186d7a2e87e30faf6fbb0bf3eda8a83a780cf19dfe809b850: Status 404 returned error can't find the container with id bc75102cce59034186d7a2e87e30faf6fbb0bf3eda8a83a780cf19dfe809b850 Oct 31 00:57:23.911715 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:23.911676 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" event={"ID":"60ad44f3-a7fe-4664-8ae5-538e73aec5fe","Type":"ContainerStarted","Data":"bc75102cce59034186d7a2e87e30faf6fbb0bf3eda8a83a780cf19dfe809b850"} Oct 31 00:57:24.925974 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:24.925942 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" event={"ID":"60ad44f3-a7fe-4664-8ae5-538e73aec5fe","Type":"ContainerStarted","Data":"335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32"} Oct 31 00:57:25.080427 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:25.080374 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" podStartSLOduration=2.080359452 podStartE2EDuration="2.080359452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:25.07840734 +0000 UTC m=+3277.553765763" watchObservedRunningTime="2026-10-31 00:57:25.080359452 +0000 UTC m=+3277.555717915" Oct 31 00:57:25.471346 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:25.471316 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:32.728796 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:32.728762 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:57:33.471894 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:33.471860 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:33.484488 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:33.484466 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:34.002711 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:34.002644 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 00:57:38.564789 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:38.564756 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7501-08240e"] Oct 31 00:57:38.569116 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:38.569097 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:38.624487 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:38.624449 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9d9vc\"" Oct 31 00:57:38.675578 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:38.675544 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7501-08240e"] Oct 31 00:57:38.685164 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:38.685129 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwv9\" (UniqueName: \"kubernetes.io/projected/6df31f81-2585-499a-ba34-9aaf1f5c7637-kube-api-access-cjwv9\") pod \"test-curl-7501-08240e\" (UID: \"6df31f81-2585-499a-ba34-9aaf1f5c7637\") " pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:38.786807 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:38.786767 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwv9\" (UniqueName: \"kubernetes.io/projected/6df31f81-2585-499a-ba34-9aaf1f5c7637-kube-api-access-cjwv9\") pod \"test-curl-7501-08240e\" (UID: \"6df31f81-2585-499a-ba34-9aaf1f5c7637\") " pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:38.848753 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:38.848674 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwv9\" (UniqueName: \"kubernetes.io/projected/6df31f81-2585-499a-ba34-9aaf1f5c7637-kube-api-access-cjwv9\") pod \"test-curl-7501-08240e\" (UID: \"6df31f81-2585-499a-ba34-9aaf1f5c7637\") " pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:38.882233 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:38.882201 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:39.259875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:39.259848 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7501-08240e"] Oct 31 00:57:39.262025 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:57:39.261998 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6df31f81_2585_499a_ba34_9aaf1f5c7637.slice/crio-f79ca38b5bf20d7fee07f4d5d149658ddf4130afe0f4b51cc40b94f58f4f9e1b WatchSource:0}: Error finding container f79ca38b5bf20d7fee07f4d5d149658ddf4130afe0f4b51cc40b94f58f4f9e1b: Status 404 returned error can't find the container with id f79ca38b5bf20d7fee07f4d5d149658ddf4130afe0f4b51cc40b94f58f4f9e1b Oct 31 00:57:40.039763 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:40.039721 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7501-08240e" event={"ID":"6df31f81-2585-499a-ba34-9aaf1f5c7637","Type":"ContainerStarted","Data":"72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012"} Oct 31 00:57:40.040163 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:40.039772 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7501-08240e" event={"ID":"6df31f81-2585-499a-ba34-9aaf1f5c7637","Type":"ContainerStarted","Data":"f79ca38b5bf20d7fee07f4d5d149658ddf4130afe0f4b51cc40b94f58f4f9e1b"} Oct 31 00:57:40.155414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:40.155361 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7501-08240e" podStartSLOduration=2.155347944 podStartE2EDuration="2.155347944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:40.152713613 +0000 UTC m=+3292.628072036" watchObservedRunningTime="2026-10-31 00:57:40.155347944 +0000 UTC m=+3292.630706368" Oct 31 00:57:41.850278 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:41.850241 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7501-08240e"] Oct 31 00:57:41.865745 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:41.865719 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7501-08240e"] Oct 31 00:57:42.047514 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.047481 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7501-188b00"] Oct 31 00:57:42.048137 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.048120 4969 cpu_manager.go:414] "containerMap: removing container" podUID="6df31f81-2585-499a-ba34-9aaf1f5c7637" containerName="test-curl-7501-08240e" Oct 31 00:57:42.048207 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.048139 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df31f81-2585-499a-ba34-9aaf1f5c7637" containerName="test-curl-7501-08240e" Oct 31 00:57:42.048267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.048258 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df31f81-2585-499a-ba34-9aaf1f5c7637" containerName="test-curl-7501-08240e" Oct 31 00:57:42.051846 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.051829 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7501-188b00" Oct 31 00:57:42.054783 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.054757 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7501-08240e" podUID="6df31f81-2585-499a-ba34-9aaf1f5c7637" containerName="test-curl-7501-08240e" containerID="cri-o://72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012" gracePeriod=2 Oct 31 00:57:42.109861 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.109792 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7501-188b00"] Oct 31 00:57:42.125847 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.125817 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v52f9\" (UniqueName: \"kubernetes.io/projected/8d4bcece-dc1b-43b8-af8f-8c5635d20873-kube-api-access-v52f9\") pod \"test-curl-7501-188b00\" (UID: \"8d4bcece-dc1b-43b8-af8f-8c5635d20873\") " pod="opendatahub/test-curl-7501-188b00" Oct 31 00:57:42.216358 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:42.216317 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7501-08240e\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="6df31f81-2585-499a-ba34-9aaf1f5c7637" pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:42.227157 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.227117 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v52f9\" (UniqueName: \"kubernetes.io/projected/8d4bcece-dc1b-43b8-af8f-8c5635d20873-kube-api-access-v52f9\") pod \"test-curl-7501-188b00\" (UID: \"8d4bcece-dc1b-43b8-af8f-8c5635d20873\") " pod="opendatahub/test-curl-7501-188b00" Oct 31 00:57:42.290121 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.290094 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v52f9\" (UniqueName: \"kubernetes.io/projected/8d4bcece-dc1b-43b8-af8f-8c5635d20873-kube-api-access-v52f9\") pod \"test-curl-7501-188b00\" (UID: \"8d4bcece-dc1b-43b8-af8f-8c5635d20873\") " pod="opendatahub/test-curl-7501-188b00" Oct 31 00:57:42.366644 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.366382 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7501-188b00" Oct 31 00:57:42.752697 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:42.752670 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7501-188b00"] Oct 31 00:57:42.753948 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:57:42.753922 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d4bcece_dc1b_43b8_af8f_8c5635d20873.slice/crio-88d5d8bc3a491d69b59678520ed9e67a9f26fc19c3cd34a523b8baa7a376c5b9 WatchSource:0}: Error finding container 88d5d8bc3a491d69b59678520ed9e67a9f26fc19c3cd34a523b8baa7a376c5b9: Status 404 returned error can't find the container with id 88d5d8bc3a491d69b59678520ed9e67a9f26fc19c3cd34a523b8baa7a376c5b9 Oct 31 00:57:43.063981 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:43.063895 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7501-188b00" event={"ID":"8d4bcece-dc1b-43b8-af8f-8c5635d20873","Type":"ContainerStarted","Data":"24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4"} Oct 31 00:57:43.063981 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:43.063936 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7501-188b00" event={"ID":"8d4bcece-dc1b-43b8-af8f-8c5635d20873","Type":"ContainerStarted","Data":"88d5d8bc3a491d69b59678520ed9e67a9f26fc19c3cd34a523b8baa7a376c5b9"} Oct 31 00:57:43.185901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:43.185845 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7501-188b00" podStartSLOduration=1.185831088 podStartE2EDuration="1.185831088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:43.18394034 +0000 UTC m=+3295.659298763" watchObservedRunningTime="2026-10-31 00:57:43.185831088 +0000 UTC m=+3295.661189511" Oct 31 00:57:43.234712 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:43.234682 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7501-08240e\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="6df31f81-2585-499a-ba34-9aaf1f5c7637" pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:44.304376 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:44.304347 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:44.453582 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:44.453552 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df31f81-2585-499a-ba34-9aaf1f5c7637-kube-api-access-cjwv9\" (UniqueName: \"kubernetes.io/projected/6df31f81-2585-499a-ba34-9aaf1f5c7637-kube-api-access-cjwv9\") pod \"6df31f81-2585-499a-ba34-9aaf1f5c7637\" (UID: \"6df31f81-2585-499a-ba34-9aaf1f5c7637\") " Oct 31 00:57:44.455875 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:44.455845 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df31f81-2585-499a-ba34-9aaf1f5c7637-kube-api-access-cjwv9" pod "6df31f81-2585-499a-ba34-9aaf1f5c7637" (UID: "6df31f81-2585-499a-ba34-9aaf1f5c7637"). InnerVolumeSpecName "kube-api-access-cjwv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:44.555733 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:44.555697 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjwv9\" (UniqueName: \"kubernetes.io/projected/6df31f81-2585-499a-ba34-9aaf1f5c7637-kube-api-access-cjwv9\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:44.977164 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:44.977123 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7501-188b00"] Oct 31 00:57:44.988499 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:44.988465 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7501-188b00"] Oct 31 00:57:45.079336 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:45.079301 4969 generic.go:356] "Generic (PLEG): container finished" podID="6df31f81-2585-499a-ba34-9aaf1f5c7637" containerID="72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012" exitCode=137 Oct 31 00:57:45.079540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:45.079352 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:45.079540 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:45.079407 4969 scope.go:122] "RemoveContainer" containerID="72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012" Oct 31 00:57:45.079719 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:45.079674 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7501-188b00" podUID="8d4bcece-dc1b-43b8-af8f-8c5635d20873" containerName="test-curl-7501-188b00" containerID="cri-o://24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4" gracePeriod=2 Oct 31 00:57:45.090391 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:45.090371 4969 scope.go:122] "RemoveContainer" containerID="72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012" Oct 31 00:57:45.090751 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:45.090705 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012\": container with ID starting with 72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012 not found: ID does not exist" containerID="72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012" Oct 31 00:57:45.090857 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:45.090758 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012"} err="failed to get container status \"72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012\": rpc error: code = NotFound desc = could not find container \"72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012\": container with ID starting with 72697256f4400695fa31b67463f4bbc121eaf8a3a166e2b63204e876a4725012 not found: ID does not exist" Oct 31 00:57:45.131543 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:45.131500 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7501-188b00\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="8d4bcece-dc1b-43b8-af8f-8c5635d20873" pod="opendatahub/test-curl-7501-188b00" Oct 31 00:57:45.182504 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:45.182468 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7501-188b00\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="8d4bcece-dc1b-43b8-af8f-8c5635d20873" pod="opendatahub/test-curl-7501-188b00" Oct 31 00:57:45.234189 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:45.234099 4969 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7501-08240e\" is forbidden: User \"system:node:ip-10-0-141-63.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-63.us-west-1.compute.internal' and this object" podUID="6df31f81-2585-499a-ba34-9aaf1f5c7637" pod="opendatahub/test-curl-7501-08240e" Oct 31 00:57:45.734881 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:45.734851 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df31f81-2585-499a-ba34-9aaf1f5c7637" path="/var/lib/kubelet/pods/6df31f81-2585-499a-ba34-9aaf1f5c7637/volumes" Oct 31 00:57:46.897398 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.897357 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx"] Oct 31 00:57:46.898236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.898210 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8d4bcece-dc1b-43b8-af8f-8c5635d20873" containerName="test-curl-7501-188b00" Oct 31 00:57:46.898347 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.898238 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4bcece-dc1b-43b8-af8f-8c5635d20873" containerName="test-curl-7501-188b00" Oct 31 00:57:46.898414 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.898389 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4bcece-dc1b-43b8-af8f-8c5635d20873" containerName="test-curl-7501-188b00" Oct 31 00:57:46.903703 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.903673 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:46.964803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.964769 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a7eb09-openshift-default-dockercfg-qsczx\"" Oct 31 00:57:46.985670 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.985639 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-socket\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:46.985815 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.985683 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-token\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:46.985815 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.985704 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istiod-ca-cert\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:46.985897 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.985816 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-credential-socket\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:46.985934 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.985892 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-certs\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:46.985972 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.985955 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rszt\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-kube-api-access-5rszt\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:46.986142 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.986120 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-envoy\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:46.986194 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.986159 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-podinfo\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:46.986332 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:46.986314 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-data\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.009352 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.009320 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx"] Oct 31 00:57:47.087408 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087376 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-socket\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.087577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087435 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-token\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.087577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087465 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istiod-ca-cert\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.087577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087493 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-credential-socket\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.087577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087526 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-certs\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.087577 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087569 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rszt\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-kube-api-access-5rszt\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.087882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087702 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-envoy\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.087882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087737 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-podinfo\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.087882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087832 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-socket\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.087882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.087862 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-data\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.088160 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.088137 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-data\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.088217 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.088164 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-certs\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.088317 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.088297 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-credential-socket\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.088371 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.088354 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istiod-ca-cert\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.090523 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.090496 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-envoy\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.090963 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.090941 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-podinfo\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.149254 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.149174 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-token\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.149731 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.149712 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rszt\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-kube-api-access-5rszt\") pod \"e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.237629 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.237577 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:47.329208 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.329059 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7501-188b00" Oct 31 00:57:47.492989 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.492905 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4bcece-dc1b-43b8-af8f-8c5635d20873-kube-api-access-v52f9\" (UniqueName: \"kubernetes.io/projected/8d4bcece-dc1b-43b8-af8f-8c5635d20873-kube-api-access-v52f9\") pod \"8d4bcece-dc1b-43b8-af8f-8c5635d20873\" (UID: \"8d4bcece-dc1b-43b8-af8f-8c5635d20873\") " Oct 31 00:57:47.495167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.495136 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4bcece-dc1b-43b8-af8f-8c5635d20873-kube-api-access-v52f9" pod "8d4bcece-dc1b-43b8-af8f-8c5635d20873" (UID: "8d4bcece-dc1b-43b8-af8f-8c5635d20873"). InnerVolumeSpecName "kube-api-access-v52f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:47.594199 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.594160 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v52f9\" (UniqueName: \"kubernetes.io/projected/8d4bcece-dc1b-43b8-af8f-8c5635d20873-kube-api-access-v52f9\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:47.685237 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.685214 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx"] Oct 31 00:57:47.686195 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:57:47.686169 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fbb87e_7f00_4ca7_8d11_2b7ec19cbab5.slice/crio-5242ba1ecc859c884f139bcef2a9445dabd1113a59511f7c2d5ab696f7aefd0c WatchSource:0}: Error finding container 5242ba1ecc859c884f139bcef2a9445dabd1113a59511f7c2d5ab696f7aefd0c: Status 404 returned error can't find the container with id 5242ba1ecc859c884f139bcef2a9445dabd1113a59511f7c2d5ab696f7aefd0c Oct 31 00:57:47.688921 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.688890 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:57:47.689029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.688956 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:57:47.689029 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.688988 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:57:47.737170 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:47.737144 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4bcece-dc1b-43b8-af8f-8c5635d20873" path="/var/lib/kubelet/pods/8d4bcece-dc1b-43b8-af8f-8c5635d20873/volumes" Oct 31 00:57:48.117626 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.117566 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" event={"ID":"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5","Type":"ContainerStarted","Data":"6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed"} Oct 31 00:57:48.118056 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.117722 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" event={"ID":"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5","Type":"ContainerStarted","Data":"5242ba1ecc859c884f139bcef2a9445dabd1113a59511f7c2d5ab696f7aefd0c"} Oct 31 00:57:48.119345 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.119318 4969 generic.go:356] "Generic (PLEG): container finished" podID="8d4bcece-dc1b-43b8-af8f-8c5635d20873" containerID="24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4" exitCode=137 Oct 31 00:57:48.119463 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.119362 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7501-188b00" Oct 31 00:57:48.119463 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.119416 4969 scope.go:122] "RemoveContainer" containerID="24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4" Oct 31 00:57:48.130369 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.130327 4969 scope.go:122] "RemoveContainer" containerID="24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4" Oct 31 00:57:48.130755 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:57:48.130726 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4\": container with ID starting with 24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4 not found: ID does not exist" containerID="24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4" Oct 31 00:57:48.130853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.130777 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4"} err="failed to get container status \"24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4\": rpc error: code = NotFound desc = could not find container \"24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4\": container with ID starting with 24d447d27537eab364f57b25d0055b91546812fcc76098b035bc7db0fe620ea4 not found: ID does not exist" Oct 31 00:57:48.239710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.239674 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:48.246511 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.246483 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:48.286708 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.286663 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" podStartSLOduration=2.286647281 podStartE2EDuration="2.286647281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:48.236380171 +0000 UTC m=+3300.711738594" watchObservedRunningTime="2026-10-31 00:57:48.286647281 +0000 UTC m=+3300.762005703" Oct 31 00:57:48.326675 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.326649 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:57:48.342702 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.342680 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:57:48.344217 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.344194 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 00:57:48.369851 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:48.369829 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 00:57:49.127375 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:49.127345 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:49.128251 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:49.128233 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:57:55.027680 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.027641 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42"] Oct 31 00:57:55.033515 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.033493 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:57:55.085710 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.085689 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-a7eb09\"" Oct 31 00:57:55.135236 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.135202 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-a7eb09-dockercfg-r7llv\"" Oct 31 00:57:55.152663 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.143538 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42"] Oct 31 00:57:55.182823 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.182793 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gsgf\" (UniqueName: \"kubernetes.io/projected/10afad44-9b46-438a-8de0-affbdb0e8405-kube-api-access-6gsgf\") pod \"payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42\" (UID: \"10afad44-9b46-438a-8de0-affbdb0e8405\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:57:55.183034 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.182863 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10afad44-9b46-438a-8de0-affbdb0e8405-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42\" (UID: \"10afad44-9b46-438a-8de0-affbdb0e8405\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:57:55.188882 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.188857 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk"] Oct 31 00:57:55.192976 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.192960 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:57:55.284172 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.284102 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gsgf\" (UniqueName: \"kubernetes.io/projected/10afad44-9b46-438a-8de0-affbdb0e8405-kube-api-access-6gsgf\") pod \"payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42\" (UID: \"10afad44-9b46-438a-8de0-affbdb0e8405\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:57:55.284333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.284183 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10afad44-9b46-438a-8de0-affbdb0e8405-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42\" (UID: \"10afad44-9b46-438a-8de0-affbdb0e8405\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:57:55.284333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.284214 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a14c40-5fed-42f8-b093-f167eefd7bbd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk\" (UID: \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:57:55.284333 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.284292 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcdhv\" (UniqueName: \"kubernetes.io/projected/a9a14c40-5fed-42f8-b093-f167eefd7bbd-kube-api-access-hcdhv\") pod \"payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk\" (UID: \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:57:55.285005 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.284979 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10afad44-9b46-438a-8de0-affbdb0e8405-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42\" (UID: \"10afad44-9b46-438a-8de0-affbdb0e8405\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:57:55.296925 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.296899 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk"] Oct 31 00:57:55.358393 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.358365 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gsgf\" (UniqueName: \"kubernetes.io/projected/10afad44-9b46-438a-8de0-affbdb0e8405-kube-api-access-6gsgf\") pod \"payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42\" (UID: \"10afad44-9b46-438a-8de0-affbdb0e8405\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:57:55.385167 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.385138 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a14c40-5fed-42f8-b093-f167eefd7bbd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk\" (UID: \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:57:55.385301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.385249 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcdhv\" (UniqueName: \"kubernetes.io/projected/a9a14c40-5fed-42f8-b093-f167eefd7bbd-kube-api-access-hcdhv\") pod \"payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk\" (UID: \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:57:55.385831 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.385812 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a14c40-5fed-42f8-b093-f167eefd7bbd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk\" (UID: \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:57:55.447878 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.447842 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcdhv\" (UniqueName: \"kubernetes.io/projected/a9a14c40-5fed-42f8-b093-f167eefd7bbd-kube-api-access-hcdhv\") pod \"payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk\" (UID: \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:57:55.506350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.506317 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:57:55.647447 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.647414 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:57:55.887091 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:57:55.887046 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a14c40_5fed_42f8_b093_f167eefd7bbd.slice/crio-17cf65ebe479c38b1dd56f6fc3c126b0075f0d7274d60f206eb0e0770ca7a01e WatchSource:0}: Error finding container 17cf65ebe479c38b1dd56f6fc3c126b0075f0d7274d60f206eb0e0770ca7a01e: Status 404 returned error can't find the container with id 17cf65ebe479c38b1dd56f6fc3c126b0075f0d7274d60f206eb0e0770ca7a01e Oct 31 00:57:55.887091 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:55.887074 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk"] Oct 31 00:57:56.029272 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:56.029245 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42"] Oct 31 00:57:56.030655 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:57:56.030570 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10afad44_9b46_438a_8de0_affbdb0e8405.slice/crio-33ec1c8fd5159c8eaa814051636521e58998cf490e230545207b893799d2bad4 WatchSource:0}: Error finding container 33ec1c8fd5159c8eaa814051636521e58998cf490e230545207b893799d2bad4: Status 404 returned error can't find the container with id 33ec1c8fd5159c8eaa814051636521e58998cf490e230545207b893799d2bad4 Oct 31 00:57:56.181411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:56.181330 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" event={"ID":"10afad44-9b46-438a-8de0-affbdb0e8405","Type":"ContainerStarted","Data":"4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29"} Oct 31 00:57:56.181411 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:56.181394 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" event={"ID":"10afad44-9b46-438a-8de0-affbdb0e8405","Type":"ContainerStarted","Data":"33ec1c8fd5159c8eaa814051636521e58998cf490e230545207b893799d2bad4"} Oct 31 00:57:56.181587 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:56.181422 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:57:56.183216 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:56.183181 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" event={"ID":"a9a14c40-5fed-42f8-b093-f167eefd7bbd","Type":"ContainerStarted","Data":"e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a"} Oct 31 00:57:56.183339 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:56.183239 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" event={"ID":"a9a14c40-5fed-42f8-b093-f167eefd7bbd","Type":"ContainerStarted","Data":"17cf65ebe479c38b1dd56f6fc3c126b0075f0d7274d60f206eb0e0770ca7a01e"} Oct 31 00:57:56.183339 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:56.183276 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:57:56.302904 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:56.302855 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" podStartSLOduration=2.302840625 podStartE2EDuration="2.302840625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:56.30153944 +0000 UTC m=+3308.776897862" watchObservedRunningTime="2026-10-31 00:57:56.302840625 +0000 UTC m=+3308.778199051" Oct 31 00:57:56.417871 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:57:56.417798 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" podStartSLOduration=1.417780937 podStartE2EDuration="1.417780937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:56.416420888 +0000 UTC m=+3308.891779312" watchObservedRunningTime="2026-10-31 00:57:56.417780937 +0000 UTC m=+3308.893139360" Oct 31 00:58:02.727816 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:58:02.727778 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:58:07.194802 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:07.194764 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 00:58:07.195302 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:07.194846 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 00:58:21.670184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.670148 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf"] Oct 31 00:58:21.674476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.674460 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.727259 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.727222 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-bf325c-openshift-default-dockercfg-ms7qj\"" Oct 31 00:58:21.781322 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.781289 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf"] Oct 31 00:58:21.789186 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.789153 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-data\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.789350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.789201 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-envoy\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.789350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.789227 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istiod-ca-cert\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.789350 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.789303 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ldj\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-kube-api-access-w2ldj\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.789510 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.789405 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-token\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.789510 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.789478 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.789780 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.789754 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-podinfo\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.789886 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.789841 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-credential-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.789938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.789887 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-certs\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.890948 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.890913 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-podinfo\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.890976 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-credential-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891008 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-certs\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891084 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-data\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891105 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-envoy\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891156 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891120 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istiod-ca-cert\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891211 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ldj\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-kube-api-access-w2ldj\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891263 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-token\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891451 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891332 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891449 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-credential-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891470 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-data\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.891712 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891549 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-certs\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.892057 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.891767 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.892175 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.892145 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istiod-ca-cert\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.893769 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.893747 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-envoy\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.894009 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.893989 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-podinfo\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.954200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.954130 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ldj\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-kube-api-access-w2ldj\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.954200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.954135 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-token\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:21.989262 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:21.989235 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:22.446902 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:58:22.446825 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5a47e4_abdc_40ef_96c8_7777e2bca4b2.slice/crio-45b23b63100c9ba965dffa32f4d20d0ac7631ad7f28fbff858077194dc3268b2 WatchSource:0}: Error finding container 45b23b63100c9ba965dffa32f4d20d0ac7631ad7f28fbff858077194dc3268b2: Status 404 returned error can't find the container with id 45b23b63100c9ba965dffa32f4d20d0ac7631ad7f28fbff858077194dc3268b2 Oct 31 00:58:22.447035 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:22.446967 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf"] Oct 31 00:58:22.449228 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:22.449197 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:58:22.449301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:22.449262 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:58:22.449301 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:22.449289 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:58:23.394617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:23.391537 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" event={"ID":"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2","Type":"ContainerStarted","Data":"08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0"} Oct 31 00:58:23.394617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:23.391625 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" event={"ID":"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2","Type":"ContainerStarted","Data":"45b23b63100c9ba965dffa32f4d20d0ac7631ad7f28fbff858077194dc3268b2"} Oct 31 00:58:23.992618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:23.992416 4969 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 00:58:23.992618 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:23.992479 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 00:58:23.992938 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:23.992799 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:24.992619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:24.990968 4969 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 00:58:24.992619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:24.991073 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 00:58:25.995617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:25.990272 4969 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 00:58:25.995617 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:25.990345 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 00:58:27.002031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:27.001991 4969 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 00:58:27.002522 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:27.002053 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 00:58:27.998619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:27.998318 4969 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Oct 31 00:58:27.998619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:27.998382 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Oct 31 00:58:29.044724 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:29.044690 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:29.045370 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:29.045331 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:29.046267 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:29.046242 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 00:58:29.229551 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:58:29.229495 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" podStartSLOduration=8.229481377 podStartE2EDuration="8.229481377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:23.60912517 +0000 UTC m=+3336.084483593" watchObservedRunningTime="2026-10-31 00:58:29.229481377 +0000 UTC m=+3341.704839871" Oct 31 00:58:32.728315 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:58:32.728280 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:59:02.012619 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.012530 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj"] Oct 31 00:59:02.017049 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.017031 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.077803 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.077776 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj"] Oct 31 00:59:02.102854 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.102822 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.103024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.102890 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29a9d25-cb21-4358-a2a8-2644ffca19e3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.103024 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.102972 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgr2f\" (UniqueName: \"kubernetes.io/projected/c29a9d25-cb21-4358-a2a8-2644ffca19e3-kube-api-access-sgr2f\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.103149 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.103083 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.122241 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.122210 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-a7eb09-serving-cert\"" Oct 31 00:59:02.203915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.203883 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgr2f\" (UniqueName: \"kubernetes.io/projected/c29a9d25-cb21-4358-a2a8-2644ffca19e3-kube-api-access-sgr2f\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.204083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.203971 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.204083 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.204078 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.204164 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:59:02.204113 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-serving-cert: secret "maas-api-e2e-shared-a-w1-a7eb09-serving-cert" not found Oct 31 00:59:02.204220 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.204117 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29a9d25-cb21-4358-a2a8-2644ffca19e3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.204220 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:59:02.204187 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls podName:c29a9d25-cb21-4358-a2a8-2644ffca19e3 nodeName:}" failed. No retries permitted until 2026-10-31 00:59:02.704164754 +0000 UTC m=+3375.179523154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls") pod "maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" (UID: "c29a9d25-cb21-4358-a2a8-2644ffca19e3") : secret "maas-api-e2e-shared-a-w1-a7eb09-serving-cert" not found Oct 31 00:59:02.204727 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.204699 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29a9d25-cb21-4358-a2a8-2644ffca19e3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.206691 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.206671 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.267627 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.267532 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgr2f\" (UniqueName: \"kubernetes.io/projected/c29a9d25-cb21-4358-a2a8-2644ffca19e3-kube-api-access-sgr2f\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.709694 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.709656 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.712342 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.712317 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:02.728937 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:59:02.728904 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:59:02.931952 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:02.931920 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:03.322467 ip-10-0-141-63 kubenswrapper[4969]: W1031 00:59:03.322424 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29a9d25_cb21_4358_a2a8_2644ffca19e3.slice/crio-6d54ebd50b28c0f57148972709a7cf8aea49ebe2a75b3cd7ba3ed2a3cd9f6520 WatchSource:0}: Error finding container 6d54ebd50b28c0f57148972709a7cf8aea49ebe2a75b3cd7ba3ed2a3cd9f6520: Status 404 returned error can't find the container with id 6d54ebd50b28c0f57148972709a7cf8aea49ebe2a75b3cd7ba3ed2a3cd9f6520 Oct 31 00:59:03.324238 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:03.324219 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:59:03.331845 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:03.331819 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj"] Oct 31 00:59:03.701873 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:03.701836 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" event={"ID":"c29a9d25-cb21-4358-a2a8-2644ffca19e3","Type":"ContainerStarted","Data":"6d54ebd50b28c0f57148972709a7cf8aea49ebe2a75b3cd7ba3ed2a3cd9f6520"} Oct 31 00:59:04.712721 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:04.712630 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" event={"ID":"c29a9d25-cb21-4358-a2a8-2644ffca19e3","Type":"ContainerStarted","Data":"9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7"} Oct 31 00:59:04.713193 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:04.712850 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:04.850261 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:04.850203 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" podStartSLOduration=2.779991855 podStartE2EDuration="3.850189232s" totalImagesPullingTime="1.070197377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:04.846640103 +0000 UTC m=+3377.321998527" watchObservedRunningTime="2026-10-31 00:59:04.850189232 +0000 UTC m=+3377.325547655" Oct 31 00:59:10.725452 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:10.725420 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 00:59:32.728001 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:59:32.727960 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 00:59:41.132978 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:41.132942 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g"] Oct 31 00:59:55.112050 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:55.111963 4969 generic.go:356] "Generic (PLEG): container finished" podID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerID="6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed" exitCode=0 Oct 31 00:59:55.112476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:55.112039 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" event={"ID":"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5","Type":"ContainerDied","Data":"6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed"} Oct 31 00:59:55.112476 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:55.112439 4969 scope.go:122] "RemoveContainer" containerID="6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed" Oct 31 00:59:55.114045 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:55.114018 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:59:55.114119 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:55.114088 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:59:55.114158 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:55.114122 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:59:56.121483 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.121447 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" event={"ID":"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5","Type":"ContainerStarted","Data":"f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0"} Oct 31 00:59:56.154457 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.154423 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" podUID="dedc0b00-d8de-4b79-8b1d-80f3dda6c964" containerName="main" containerID="cri-o://182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86" gracePeriod=45 Oct 31 00:59:56.238544 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.238516 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 00:59:56.238760 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.238680 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:59:56.238760 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.238744 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 31 00:59:56.248235 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.248192 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" podUID="dedc0b00-d8de-4b79-8b1d-80f3dda6c964" containerName="main" probeResult="failure" output="Get \"https://10.133.0.227:8000/ready\": dial tcp 10.133.0.227:8000: connect: connection refused" Oct 31 00:59:56.523443 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.523416 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:59:56.612324 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.612288 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tls-certs\" (UniqueName: \"kubernetes.io/secret/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tls-certs\") pod \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " Oct 31 00:59:56.612548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.612336 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-model-cache\") pod \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " Oct 31 00:59:56.612548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.612404 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-dshm\") pod \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " Oct 31 00:59:56.612548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.612445 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tmp-dir\") pod \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " Oct 31 00:59:56.612548 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.612507 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-home\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-home\") pod \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " Oct 31 00:59:56.612853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.612625 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-model-cache" pod "dedc0b00-d8de-4b79-8b1d-80f3dda6c964" (UID: "dedc0b00-d8de-4b79-8b1d-80f3dda6c964"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:56.612853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.612646 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-kube-api-access-grbjn\" (UniqueName: \"kubernetes.io/projected/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-kube-api-access-grbjn\") pod \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\" (UID: \"dedc0b00-d8de-4b79-8b1d-80f3dda6c964\") " Oct 31 00:59:56.612853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.612726 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tmp-dir" pod "dedc0b00-d8de-4b79-8b1d-80f3dda6c964" (UID: "dedc0b00-d8de-4b79-8b1d-80f3dda6c964"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:56.612853 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.612767 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-home" pod "dedc0b00-d8de-4b79-8b1d-80f3dda6c964" (UID: "dedc0b00-d8de-4b79-8b1d-80f3dda6c964"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:56.613245 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.613225 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:56.613337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.613270 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:56.613337 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.613283 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:56.615062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.615035 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tls-certs" pod "dedc0b00-d8de-4b79-8b1d-80f3dda6c964" (UID: "dedc0b00-d8de-4b79-8b1d-80f3dda6c964"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:56.615062 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.615048 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-dshm" pod "dedc0b00-d8de-4b79-8b1d-80f3dda6c964" (UID: "dedc0b00-d8de-4b79-8b1d-80f3dda6c964"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:56.615200 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.615094 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-kube-api-access-grbjn" pod "dedc0b00-d8de-4b79-8b1d-80f3dda6c964" (UID: "dedc0b00-d8de-4b79-8b1d-80f3dda6c964"). InnerVolumeSpecName "kube-api-access-grbjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:56.715343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.715303 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:56.715343 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.715347 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:56.715561 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:56.715362 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grbjn\" (UniqueName: \"kubernetes.io/projected/dedc0b00-d8de-4b79-8b1d-80f3dda6c964-kube-api-access-grbjn\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.131728 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.131692 4969 generic.go:356] "Generic (PLEG): container finished" podID="dedc0b00-d8de-4b79-8b1d-80f3dda6c964" containerID="182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86" exitCode=0 Oct 31 00:59:57.132184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.131740 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" Oct 31 00:59:57.132184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.131777 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" event={"ID":"dedc0b00-d8de-4b79-8b1d-80f3dda6c964","Type":"ContainerDied","Data":"182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86"} Oct 31 00:59:57.132184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.131826 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g" event={"ID":"dedc0b00-d8de-4b79-8b1d-80f3dda6c964","Type":"ContainerDied","Data":"7ae8329fb18c09d8ae043e9890ef18eb8f1c25ea2d123c478b171274339bed49"} Oct 31 00:59:57.132184 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.131847 4969 scope.go:122] "RemoveContainer" containerID="182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86" Oct 31 00:59:57.144031 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.144009 4969 scope.go:122] "RemoveContainer" containerID="182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86" Oct 31 00:59:57.144345 ip-10-0-141-63 kubenswrapper[4969]: E1031 00:59:57.144309 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86\": container with ID starting with 182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86 not found: ID does not exist" containerID="182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86" Oct 31 00:59:57.144684 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.144352 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86"} err="failed to get container status \"182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86\": rpc error: code = NotFound desc = could not find container \"182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86\": container with ID starting with 182f1bab783bfbea7df8c1d5ca8310bd482bb8fed94c331bead63a4dc3158e86 not found: ID does not exist" Oct 31 00:59:57.238696 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.238665 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:59:57.238848 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.238712 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 31 00:59:57.312406 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.312373 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g"] Oct 31 00:59:57.353410 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.353376 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cfc055/auth-test-model-cfc055-kserve-5dd8596657-zwf8g"] Oct 31 00:59:57.738543 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:57.738510 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dedc0b00-d8de-4b79-8b1d-80f3dda6c964" path="/var/lib/kubelet/pods/dedc0b00-d8de-4b79-8b1d-80f3dda6c964/volumes" Oct 31 00:59:58.238390 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:58.238348 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:59:58.238901 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:58.238416 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 31 00:59:59.238392 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:59.238350 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 31 00:59:59.238915 ip-10-0-141-63 kubenswrapper[4969]: I1031 00:59:59.238412 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 31 01:00:00.239424 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.235562 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv"] Oct 31 01:00:00.239424 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.236913 4969 cpu_manager.go:414] "containerMap: removing container" podUID="dedc0b00-d8de-4b79-8b1d-80f3dda6c964" containerName="main" Oct 31 01:00:00.239424 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.236940 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dedc0b00-d8de-4b79-8b1d-80f3dda6c964" containerName="main" Oct 31 01:00:00.239424 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.237306 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="dedc0b00-d8de-4b79-8b1d-80f3dda6c964" containerName="main" Oct 31 01:00:00.240126 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.239733 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 31 01:00:00.240126 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.239826 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 31 01:00:00.243860 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.243835 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:00.252637 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.252613 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7"] Oct 31 01:00:00.257196 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.257175 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv"] Oct 31 01:00:00.257304 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.257270 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:00.261657 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.261635 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:00.308722 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.308693 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj"] Oct 31 01:00:00.313224 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.313152 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng"] Oct 31 01:00:00.313395 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.313272 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:00.317862 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.317840 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:00.361104 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361074 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58a02d2-f107-4ce3-a2e8-680eb5583abc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7\" (UID: \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:00.361312 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361290 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea7dc39-a164-44d7-8959-ab89867c8100-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-mmpsj\" (UID: \"2ea7dc39-a164-44d7-8959-ab89867c8100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:00.361429 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361395 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7jhf\" (UniqueName: \"kubernetes.io/projected/2ea7dc39-a164-44d7-8959-ab89867c8100-kube-api-access-k7jhf\") pod \"maas-api-key-cleanup-29890140-mmpsj\" (UID: \"2ea7dc39-a164-44d7-8959-ab89867c8100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:00.361475 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361423 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wm4\" (UniqueName: \"kubernetes.io/projected/c34afbdf-0019-4945-ae98-076e3faab54c-kube-api-access-d5wm4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng\" (UID: \"c34afbdf-0019-4945-ae98-076e3faab54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:00.361475 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361459 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d406a-111c-4af4-95ba-b7e0304c399f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv\" (UID: \"ed2d406a-111c-4af4-95ba-b7e0304c399f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:00.361545 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361507 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kggrt\" (UniqueName: \"kubernetes.io/projected/ed2d406a-111c-4af4-95ba-b7e0304c399f-kube-api-access-kggrt\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv\" (UID: \"ed2d406a-111c-4af4-95ba-b7e0304c399f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:00.361545 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361524 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qc8q\" (UniqueName: \"kubernetes.io/projected/d58a02d2-f107-4ce3-a2e8-680eb5583abc-kube-api-access-9qc8q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7\" (UID: \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:00.361632 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361548 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4813544-394e-4440-af14-a7b2a3c520be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv\" (UID: \"e4813544-394e-4440-af14-a7b2a3c520be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:00.361632 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361579 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34afbdf-0019-4945-ae98-076e3faab54c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng\" (UID: \"c34afbdf-0019-4945-ae98-076e3faab54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:00.361632 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.361624 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlxp\" (UniqueName: \"kubernetes.io/projected/e4813544-394e-4440-af14-a7b2a3c520be-kube-api-access-tvlxp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv\" (UID: \"e4813544-394e-4440-af14-a7b2a3c520be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:00.363630 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.362802 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv"] Oct 31 01:00:00.363630 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.363226 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:00:00.369470 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.367259 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7"] Oct 31 01:00:00.423152 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.422765 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv"] Oct 31 01:00:00.426705 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.426674 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj"] Oct 31 01:00:00.431956 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.431930 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng"] Oct 31 01:00:00.462384 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462357 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7jhf\" (UniqueName: \"kubernetes.io/projected/2ea7dc39-a164-44d7-8959-ab89867c8100-kube-api-access-k7jhf\") pod \"maas-api-key-cleanup-29890140-mmpsj\" (UID: \"2ea7dc39-a164-44d7-8959-ab89867c8100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:00.462546 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462390 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wm4\" (UniqueName: \"kubernetes.io/projected/c34afbdf-0019-4945-ae98-076e3faab54c-kube-api-access-d5wm4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng\" (UID: \"c34afbdf-0019-4945-ae98-076e3faab54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:00.462546 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462422 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d406a-111c-4af4-95ba-b7e0304c399f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv\" (UID: \"ed2d406a-111c-4af4-95ba-b7e0304c399f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:00.462546 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462456 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kggrt\" (UniqueName: \"kubernetes.io/projected/ed2d406a-111c-4af4-95ba-b7e0304c399f-kube-api-access-kggrt\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv\" (UID: \"ed2d406a-111c-4af4-95ba-b7e0304c399f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:00.462546 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462474 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qc8q\" (UniqueName: \"kubernetes.io/projected/d58a02d2-f107-4ce3-a2e8-680eb5583abc-kube-api-access-9qc8q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7\" (UID: \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:00.462546 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462499 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4813544-394e-4440-af14-a7b2a3c520be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv\" (UID: \"e4813544-394e-4440-af14-a7b2a3c520be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:00.462546 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462534 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34afbdf-0019-4945-ae98-076e3faab54c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng\" (UID: \"c34afbdf-0019-4945-ae98-076e3faab54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:00.463051 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462633 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlxp\" (UniqueName: \"kubernetes.io/projected/e4813544-394e-4440-af14-a7b2a3c520be-kube-api-access-tvlxp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv\" (UID: \"e4813544-394e-4440-af14-a7b2a3c520be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:00.463051 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462762 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58a02d2-f107-4ce3-a2e8-680eb5583abc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7\" (UID: \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:00.463051 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.462816 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea7dc39-a164-44d7-8959-ab89867c8100-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-mmpsj\" (UID: \"2ea7dc39-a164-44d7-8959-ab89867c8100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:00.463404 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.463377 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4813544-394e-4440-af14-a7b2a3c520be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv\" (UID: \"e4813544-394e-4440-af14-a7b2a3c520be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:00.463404 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.463398 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58a02d2-f107-4ce3-a2e8-680eb5583abc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7\" (UID: \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:00.463513 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.463428 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d406a-111c-4af4-95ba-b7e0304c399f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv\" (UID: \"ed2d406a-111c-4af4-95ba-b7e0304c399f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:00.463513 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.463456 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea7dc39-a164-44d7-8959-ab89867c8100-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-mmpsj\" (UID: \"2ea7dc39-a164-44d7-8959-ab89867c8100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:00.463513 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.463428 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34afbdf-0019-4945-ae98-076e3faab54c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng\" (UID: \"c34afbdf-0019-4945-ae98-076e3faab54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:00.530080 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.530014 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlxp\" (UniqueName: \"kubernetes.io/projected/e4813544-394e-4440-af14-a7b2a3c520be-kube-api-access-tvlxp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv\" (UID: \"e4813544-394e-4440-af14-a7b2a3c520be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:00.530080 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.530014 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wm4\" (UniqueName: \"kubernetes.io/projected/c34afbdf-0019-4945-ae98-076e3faab54c-kube-api-access-d5wm4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng\" (UID: \"c34afbdf-0019-4945-ae98-076e3faab54c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:00.530551 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.530533 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kggrt\" (UniqueName: \"kubernetes.io/projected/ed2d406a-111c-4af4-95ba-b7e0304c399f-kube-api-access-kggrt\") pod \"maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv\" (UID: \"ed2d406a-111c-4af4-95ba-b7e0304c399f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:00.531306 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.531273 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qc8q\" (UniqueName: \"kubernetes.io/projected/d58a02d2-f107-4ce3-a2e8-680eb5583abc-kube-api-access-9qc8q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7\" (UID: \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:00.532295 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.532272 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7jhf\" (UniqueName: \"kubernetes.io/projected/2ea7dc39-a164-44d7-8959-ab89867c8100-kube-api-access-k7jhf\") pod \"maas-api-key-cleanup-29890140-mmpsj\" (UID: \"2ea7dc39-a164-44d7-8959-ab89867c8100\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:00.561238 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.561217 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:00.571689 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.571669 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:00.606002 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.605954 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:00.629913 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.629798 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:00.635975 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:00.635950 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:01.012523 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:00:01.012473 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4813544_394e_4440_af14_a7b2a3c520be.slice/crio-4f66e8d6847d6430612cda2b7422d190ccdc8599fe0eb730b491017b4a57af76 WatchSource:0}: Error finding container 4f66e8d6847d6430612cda2b7422d190ccdc8599fe0eb730b491017b4a57af76: Status 404 returned error can't find the container with id 4f66e8d6847d6430612cda2b7422d190ccdc8599fe0eb730b491017b4a57af76 Oct 31 01:00:01.013066 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.013040 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv"] Oct 31 01:00:01.035289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.035261 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7"] Oct 31 01:00:01.036042 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:00:01.036018 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58a02d2_f107_4ce3_a2e8_680eb5583abc.slice/crio-a5ed520b79a92a857f93b84b38bb7a4ae7c9e6f9075c7b037039cdc6edfdf23a WatchSource:0}: Error finding container a5ed520b79a92a857f93b84b38bb7a4ae7c9e6f9075c7b037039cdc6edfdf23a: Status 404 returned error can't find the container with id a5ed520b79a92a857f93b84b38bb7a4ae7c9e6f9075c7b037039cdc6edfdf23a Oct 31 01:00:01.087965 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.087930 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv"] Oct 31 01:00:01.149007 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.148984 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng"] Oct 31 01:00:01.166837 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.166804 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj"] Oct 31 01:00:01.169624 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.169578 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" event={"ID":"d58a02d2-f107-4ce3-a2e8-680eb5583abc","Type":"ContainerStarted","Data":"a5ed520b79a92a857f93b84b38bb7a4ae7c9e6f9075c7b037039cdc6edfdf23a"} Oct 31 01:00:01.171422 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.171384 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" event={"ID":"ed2d406a-111c-4af4-95ba-b7e0304c399f","Type":"ContainerStarted","Data":"ec3dd26456d0735c78a6928d14a184effcdcd7d83830132d603ae992eae6b296"} Oct 31 01:00:01.173565 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.173257 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" event={"ID":"e4813544-394e-4440-af14-a7b2a3c520be","Type":"ContainerStarted","Data":"4f66e8d6847d6430612cda2b7422d190ccdc8599fe0eb730b491017b4a57af76"} Oct 31 01:00:01.305362 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.305256 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 01:00:01.305830 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.305782 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 01:00:01.306792 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:01.306774 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 01:00:02.186644 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.186572 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" event={"ID":"e4813544-394e-4440-af14-a7b2a3c520be","Type":"ContainerStarted","Data":"77748a6a74c9759191a6a0ca2d1fdf44dd3691327a490b580ad12388b423e690"} Oct 31 01:00:02.188820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.188779 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" event={"ID":"d58a02d2-f107-4ce3-a2e8-680eb5583abc","Type":"ContainerStarted","Data":"275cae804f02849ba6a5d9d5d3d1c93fc99986bac42480aa34e793c51e9af265"} Oct 31 01:00:02.190888 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.190863 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" event={"ID":"ed2d406a-111c-4af4-95ba-b7e0304c399f","Type":"ContainerStarted","Data":"2c5d2f3f721a356c9366197dc257d9d68185278dc2fd2d69455a18d0ef29e0b3"} Oct 31 01:00:02.191360 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.191149 4969 scope.go:122] "RemoveContainer" containerID="2c5d2f3f721a356c9366197dc257d9d68185278dc2fd2d69455a18d0ef29e0b3" Oct 31 01:00:02.193604 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.193565 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" event={"ID":"2ea7dc39-a164-44d7-8959-ab89867c8100","Type":"ContainerStarted","Data":"06ebf93c12cd83194cb861229176b6daa315b36c34d7715740109a5d6c24c87f"} Oct 31 01:00:02.193710 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.193624 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" event={"ID":"2ea7dc39-a164-44d7-8959-ab89867c8100","Type":"ContainerStarted","Data":"cff064ee445069769a3ae94102780523c01e74a9b0ab995897903781e8360a42"} Oct 31 01:00:02.196155 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.196128 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" event={"ID":"c34afbdf-0019-4945-ae98-076e3faab54c","Type":"ContainerStarted","Data":"12af5eeb08d895a0761e8e94b1d0b39e75f06ec4e1f5778ce7984368b22376f1"} Oct 31 01:00:02.196250 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.196179 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" event={"ID":"c34afbdf-0019-4945-ae98-076e3faab54c","Type":"ContainerStarted","Data":"9f21a8eb01e9e8e71d58cbeac510966a5b9db9832ca0a4adbca7e5da784818eb"} Oct 31 01:00:02.302770 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.302709 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" podStartSLOduration=2.302691469 podStartE2EDuration="2.302691469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.301868935 +0000 UTC m=+3434.777227362" watchObservedRunningTime="2026-10-31 01:00:02.302691469 +0000 UTC m=+3434.778049892" Oct 31 01:00:02.427489 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.427437 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" podStartSLOduration=2.427422623 podStartE2EDuration="2.427422623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.42236935 +0000 UTC m=+3434.897727772" watchObservedRunningTime="2026-10-31 01:00:02.427422623 +0000 UTC m=+3434.902781043" Oct 31 01:00:02.540033 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.539975 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" podStartSLOduration=2.539959514 podStartE2EDuration="2.539959514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.537076235 +0000 UTC m=+3435.012434659" watchObservedRunningTime="2026-10-31 01:00:02.539959514 +0000 UTC m=+3435.015317937" Oct 31 01:00:02.728848 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:02.728805 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:00:02.769440 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:02.769373 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" podStartSLOduration=2.769353358 podStartE2EDuration="2.769353358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.76614946 +0000 UTC m=+3435.241507883" watchObservedRunningTime="2026-10-31 01:00:02.769353358 +0000 UTC m=+3435.244711786" Oct 31 01:00:03.205103 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.205061 4969 generic.go:356] "Generic (PLEG): container finished" podID="c34afbdf-0019-4945-ae98-076e3faab54c" containerID="12af5eeb08d895a0761e8e94b1d0b39e75f06ec4e1f5778ce7984368b22376f1" exitCode=60 Oct 31 01:00:03.205250 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.205148 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" event={"ID":"c34afbdf-0019-4945-ae98-076e3faab54c","Type":"ContainerDied","Data":"12af5eeb08d895a0761e8e94b1d0b39e75f06ec4e1f5778ce7984368b22376f1"} Oct 31 01:00:03.205517 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.205491 4969 scope.go:122] "RemoveContainer" containerID="12af5eeb08d895a0761e8e94b1d0b39e75f06ec4e1f5778ce7984368b22376f1" Oct 31 01:00:03.207041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.207019 4969 generic.go:356] "Generic (PLEG): container finished" podID="e4813544-394e-4440-af14-a7b2a3c520be" containerID="77748a6a74c9759191a6a0ca2d1fdf44dd3691327a490b580ad12388b423e690" exitCode=60 Oct 31 01:00:03.207161 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.207104 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" event={"ID":"e4813544-394e-4440-af14-a7b2a3c520be","Type":"ContainerDied","Data":"77748a6a74c9759191a6a0ca2d1fdf44dd3691327a490b580ad12388b423e690"} Oct 31 01:00:03.207332 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.207293 4969 scope.go:122] "RemoveContainer" containerID="77748a6a74c9759191a6a0ca2d1fdf44dd3691327a490b580ad12388b423e690" Oct 31 01:00:03.209172 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.209104 4969 generic.go:356] "Generic (PLEG): container finished" podID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerID="275cae804f02849ba6a5d9d5d3d1c93fc99986bac42480aa34e793c51e9af265" exitCode=60 Oct 31 01:00:03.209252 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.209196 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" event={"ID":"d58a02d2-f107-4ce3-a2e8-680eb5583abc","Type":"ContainerDied","Data":"275cae804f02849ba6a5d9d5d3d1c93fc99986bac42480aa34e793c51e9af265"} Oct 31 01:00:03.209447 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.209429 4969 scope.go:122] "RemoveContainer" containerID="275cae804f02849ba6a5d9d5d3d1c93fc99986bac42480aa34e793c51e9af265" Oct 31 01:00:03.211472 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.211447 4969 generic.go:356] "Generic (PLEG): container finished" podID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerID="2c5d2f3f721a356c9366197dc257d9d68185278dc2fd2d69455a18d0ef29e0b3" exitCode=60 Oct 31 01:00:03.211582 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.211490 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" event={"ID":"ed2d406a-111c-4af4-95ba-b7e0304c399f","Type":"ContainerDied","Data":"2c5d2f3f721a356c9366197dc257d9d68185278dc2fd2d69455a18d0ef29e0b3"} Oct 31 01:00:03.211582 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.211548 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" event={"ID":"ed2d406a-111c-4af4-95ba-b7e0304c399f","Type":"ContainerStarted","Data":"80664b705577cde853d771a766195e1fdc17c81d75a19cd0e62dd265ac1c7dae"} Oct 31 01:00:03.213395 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.213370 4969 generic.go:356] "Generic (PLEG): container finished" podID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerID="06ebf93c12cd83194cb861229176b6daa315b36c34d7715740109a5d6c24c87f" exitCode=60 Oct 31 01:00:03.213482 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.213446 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" event={"ID":"2ea7dc39-a164-44d7-8959-ab89867c8100","Type":"ContainerDied","Data":"06ebf93c12cd83194cb861229176b6daa315b36c34d7715740109a5d6c24c87f"} Oct 31 01:00:03.213701 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.213684 4969 scope.go:122] "RemoveContainer" containerID="06ebf93c12cd83194cb861229176b6daa315b36c34d7715740109a5d6c24c87f" Oct 31 01:00:03.445843 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:03.445738 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" podStartSLOduration=3.445724492 podStartE2EDuration="3.445724492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:03.441484802 +0000 UTC m=+3435.916843226" watchObservedRunningTime="2026-10-31 01:00:03.445724492 +0000 UTC m=+3435.921082914" Oct 31 01:00:04.223240 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.223207 4969 generic.go:356] "Generic (PLEG): container finished" podID="c34afbdf-0019-4945-ae98-076e3faab54c" containerID="69e43c143f9c7ecf1d112740daa37c35449f6b9c5bd577832ce303c5e4318f4d" exitCode=60 Oct 31 01:00:04.223423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.223292 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" event={"ID":"c34afbdf-0019-4945-ae98-076e3faab54c","Type":"ContainerDied","Data":"69e43c143f9c7ecf1d112740daa37c35449f6b9c5bd577832ce303c5e4318f4d"} Oct 31 01:00:04.223423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.223362 4969 scope.go:122] "RemoveContainer" containerID="12af5eeb08d895a0761e8e94b1d0b39e75f06ec4e1f5778ce7984368b22376f1" Oct 31 01:00:04.223676 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.223662 4969 scope.go:122] "RemoveContainer" containerID="69e43c143f9c7ecf1d112740daa37c35449f6b9c5bd577832ce303c5e4318f4d" Oct 31 01:00:04.224010 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:04.223988 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng_odh-ai-gateway-infra(c34afbdf-0019-4945-ae98-076e3faab54c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" Oct 31 01:00:04.225498 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.225470 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" event={"ID":"e4813544-394e-4440-af14-a7b2a3c520be","Type":"ContainerStarted","Data":"8498623e6e349fa017fa19640c449cc4bf455ef5a6e4a7ab4c7e192fd916f856"} Oct 31 01:00:04.227659 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.227622 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" event={"ID":"d58a02d2-f107-4ce3-a2e8-680eb5583abc","Type":"ContainerStarted","Data":"474cc21a18b43adfe1fa3a49b1efc816beffeed3ced4c5263ce6ed721afba438"} Oct 31 01:00:04.230037 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.230011 4969 generic.go:356] "Generic (PLEG): container finished" podID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerID="80664b705577cde853d771a766195e1fdc17c81d75a19cd0e62dd265ac1c7dae" exitCode=60 Oct 31 01:00:04.230161 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.230137 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" event={"ID":"ed2d406a-111c-4af4-95ba-b7e0304c399f","Type":"ContainerDied","Data":"80664b705577cde853d771a766195e1fdc17c81d75a19cd0e62dd265ac1c7dae"} Oct 31 01:00:04.230357 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.230337 4969 scope.go:122] "RemoveContainer" containerID="80664b705577cde853d771a766195e1fdc17c81d75a19cd0e62dd265ac1c7dae" Oct 31 01:00:04.230648 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:04.230579 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv_odh-ai-gateway-infra(ed2d406a-111c-4af4-95ba-b7e0304c399f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" Oct 31 01:00:04.232186 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.232165 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" event={"ID":"2ea7dc39-a164-44d7-8959-ab89867c8100","Type":"ContainerStarted","Data":"4d374d4c5259b75d27c3113d854d53d149f6c180a712483d676a362495223f8e"} Oct 31 01:00:04.246622 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:04.246569 4969 scope.go:122] "RemoveContainer" containerID="2c5d2f3f721a356c9366197dc257d9d68185278dc2fd2d69455a18d0ef29e0b3" Oct 31 01:00:05.243345 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.243310 4969 generic.go:356] "Generic (PLEG): container finished" podID="e4813544-394e-4440-af14-a7b2a3c520be" containerID="8498623e6e349fa017fa19640c449cc4bf455ef5a6e4a7ab4c7e192fd916f856" exitCode=60 Oct 31 01:00:05.243889 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.243395 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" event={"ID":"e4813544-394e-4440-af14-a7b2a3c520be","Type":"ContainerDied","Data":"8498623e6e349fa017fa19640c449cc4bf455ef5a6e4a7ab4c7e192fd916f856"} Oct 31 01:00:05.243889 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.243470 4969 scope.go:122] "RemoveContainer" containerID="77748a6a74c9759191a6a0ca2d1fdf44dd3691327a490b580ad12388b423e690" Oct 31 01:00:05.243889 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.243815 4969 scope.go:122] "RemoveContainer" containerID="8498623e6e349fa017fa19640c449cc4bf455ef5a6e4a7ab4c7e192fd916f856" Oct 31 01:00:05.244203 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:05.244172 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv_odh-ai-gateway-infra(e4813544-394e-4440-af14-a7b2a3c520be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" podUID="e4813544-394e-4440-af14-a7b2a3c520be" Oct 31 01:00:05.245904 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.245880 4969 generic.go:356] "Generic (PLEG): container finished" podID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerID="474cc21a18b43adfe1fa3a49b1efc816beffeed3ced4c5263ce6ed721afba438" exitCode=60 Oct 31 01:00:05.246031 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.245967 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" event={"ID":"d58a02d2-f107-4ce3-a2e8-680eb5583abc","Type":"ContainerDied","Data":"474cc21a18b43adfe1fa3a49b1efc816beffeed3ced4c5263ce6ed721afba438"} Oct 31 01:00:05.246292 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.246276 4969 scope.go:122] "RemoveContainer" containerID="474cc21a18b43adfe1fa3a49b1efc816beffeed3ced4c5263ce6ed721afba438" Oct 31 01:00:05.246550 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:05.246532 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7_odh-ai-gateway-infra(d58a02d2-f107-4ce3-a2e8-680eb5583abc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" Oct 31 01:00:05.248740 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.248718 4969 scope.go:122] "RemoveContainer" containerID="80664b705577cde853d771a766195e1fdc17c81d75a19cd0e62dd265ac1c7dae" Oct 31 01:00:05.249074 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:05.249036 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv_odh-ai-gateway-infra(ed2d406a-111c-4af4-95ba-b7e0304c399f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" Oct 31 01:00:05.250518 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.250498 4969 generic.go:356] "Generic (PLEG): container finished" podID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerID="4d374d4c5259b75d27c3113d854d53d149f6c180a712483d676a362495223f8e" exitCode=60 Oct 31 01:00:05.250643 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.250604 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" event={"ID":"2ea7dc39-a164-44d7-8959-ab89867c8100","Type":"ContainerDied","Data":"4d374d4c5259b75d27c3113d854d53d149f6c180a712483d676a362495223f8e"} Oct 31 01:00:05.250832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.250816 4969 scope.go:122] "RemoveContainer" containerID="4d374d4c5259b75d27c3113d854d53d149f6c180a712483d676a362495223f8e" Oct 31 01:00:05.251103 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:05.251084 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-mmpsj_odh-ai-gateway-infra(2ea7dc39-a164-44d7-8959-ab89867c8100)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" Oct 31 01:00:05.252496 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.252480 4969 scope.go:122] "RemoveContainer" containerID="69e43c143f9c7ecf1d112740daa37c35449f6b9c5bd577832ce303c5e4318f4d" Oct 31 01:00:05.252768 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:05.252747 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng_odh-ai-gateway-infra(c34afbdf-0019-4945-ae98-076e3faab54c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" Oct 31 01:00:05.258205 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.258188 4969 scope.go:122] "RemoveContainer" containerID="275cae804f02849ba6a5d9d5d3d1c93fc99986bac42480aa34e793c51e9af265" Oct 31 01:00:05.276711 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:05.276689 4969 scope.go:122] "RemoveContainer" containerID="06ebf93c12cd83194cb861229176b6daa315b36c34d7715740109a5d6c24c87f" Oct 31 01:00:06.262021 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:06.261993 4969 scope.go:122] "RemoveContainer" containerID="8498623e6e349fa017fa19640c449cc4bf455ef5a6e4a7ab4c7e192fd916f856" Oct 31 01:00:06.262468 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:06.262350 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv_odh-ai-gateway-infra(e4813544-394e-4440-af14-a7b2a3c520be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" podUID="e4813544-394e-4440-af14-a7b2a3c520be" Oct 31 01:00:06.263812 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:06.263786 4969 scope.go:122] "RemoveContainer" containerID="474cc21a18b43adfe1fa3a49b1efc816beffeed3ced4c5263ce6ed721afba438" Oct 31 01:00:06.264107 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:06.264085 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7_odh-ai-gateway-infra(d58a02d2-f107-4ce3-a2e8-680eb5583abc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" Oct 31 01:00:06.265421 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:06.265404 4969 scope.go:122] "RemoveContainer" containerID="4d374d4c5259b75d27c3113d854d53d149f6c180a712483d676a362495223f8e" Oct 31 01:00:06.265631 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:06.265579 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-mmpsj_odh-ai-gateway-infra(2ea7dc39-a164-44d7-8959-ab89867c8100)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" Oct 31 01:00:15.727161 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:15.727131 4969 scope.go:122] "RemoveContainer" containerID="69e43c143f9c7ecf1d112740daa37c35449f6b9c5bd577832ce303c5e4318f4d" Oct 31 01:00:15.727563 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:15.727237 4969 scope.go:122] "RemoveContainer" containerID="80664b705577cde853d771a766195e1fdc17c81d75a19cd0e62dd265ac1c7dae" Oct 31 01:00:16.354626 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:16.354520 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" event={"ID":"c34afbdf-0019-4945-ae98-076e3faab54c","Type":"ContainerStarted","Data":"5573f11883b78748e61e785a0aa1194561c1fee7b1e72b1cab64fa303cd02800"} Oct 31 01:00:16.356652 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:16.356623 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" event={"ID":"ed2d406a-111c-4af4-95ba-b7e0304c399f","Type":"ContainerStarted","Data":"f50d2205e077c9cebe783ac227ad13998630ebe31a32d7dfb9cfde11af60e4ac"} Oct 31 01:00:16.726821 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:16.726787 4969 scope.go:122] "RemoveContainer" containerID="8498623e6e349fa017fa19640c449cc4bf455ef5a6e4a7ab4c7e192fd916f856" Oct 31 01:00:17.366390 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.366357 4969 generic.go:356] "Generic (PLEG): container finished" podID="c34afbdf-0019-4945-ae98-076e3faab54c" containerID="5573f11883b78748e61e785a0aa1194561c1fee7b1e72b1cab64fa303cd02800" exitCode=60 Oct 31 01:00:17.366859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.366431 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" event={"ID":"c34afbdf-0019-4945-ae98-076e3faab54c","Type":"ContainerDied","Data":"5573f11883b78748e61e785a0aa1194561c1fee7b1e72b1cab64fa303cd02800"} Oct 31 01:00:17.366859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.366492 4969 scope.go:122] "RemoveContainer" containerID="69e43c143f9c7ecf1d112740daa37c35449f6b9c5bd577832ce303c5e4318f4d" Oct 31 01:00:17.366973 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.366943 4969 scope.go:122] "RemoveContainer" containerID="5573f11883b78748e61e785a0aa1194561c1fee7b1e72b1cab64fa303cd02800" Oct 31 01:00:17.367352 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:17.367322 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng_odh-ai-gateway-infra(c34afbdf-0019-4945-ae98-076e3faab54c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" Oct 31 01:00:17.368920 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.368895 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" event={"ID":"e4813544-394e-4440-af14-a7b2a3c520be","Type":"ContainerStarted","Data":"4ebd8142c891b5bcd532ec06f76c90fb115678295d1a5a87eee47442bb497dcb"} Oct 31 01:00:17.370750 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.370729 4969 generic.go:356] "Generic (PLEG): container finished" podID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerID="f50d2205e077c9cebe783ac227ad13998630ebe31a32d7dfb9cfde11af60e4ac" exitCode=60 Oct 31 01:00:17.370843 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.370788 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" event={"ID":"ed2d406a-111c-4af4-95ba-b7e0304c399f","Type":"ContainerDied","Data":"f50d2205e077c9cebe783ac227ad13998630ebe31a32d7dfb9cfde11af60e4ac"} Oct 31 01:00:17.371126 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.371111 4969 scope.go:122] "RemoveContainer" containerID="f50d2205e077c9cebe783ac227ad13998630ebe31a32d7dfb9cfde11af60e4ac" Oct 31 01:00:17.371400 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:17.371383 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv_odh-ai-gateway-infra(ed2d406a-111c-4af4-95ba-b7e0304c399f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" Oct 31 01:00:17.380648 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.380629 4969 scope.go:122] "RemoveContainer" containerID="80664b705577cde853d771a766195e1fdc17c81d75a19cd0e62dd265ac1c7dae" Oct 31 01:00:17.482855 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.482822 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng"] Oct 31 01:00:17.599937 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.599911 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv"] Oct 31 01:00:17.734540 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.734461 4969 scope.go:122] "RemoveContainer" containerID="4d374d4c5259b75d27c3113d854d53d149f6c180a712483d676a362495223f8e" Oct 31 01:00:17.734714 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:17.734586 4969 scope.go:122] "RemoveContainer" containerID="474cc21a18b43adfe1fa3a49b1efc816beffeed3ced4c5263ce6ed721afba438" Oct 31 01:00:18.380151 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.380119 4969 generic.go:356] "Generic (PLEG): container finished" podID="e4813544-394e-4440-af14-a7b2a3c520be" containerID="4ebd8142c891b5bcd532ec06f76c90fb115678295d1a5a87eee47442bb497dcb" exitCode=60 Oct 31 01:00:18.380651 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.380189 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" event={"ID":"e4813544-394e-4440-af14-a7b2a3c520be","Type":"ContainerDied","Data":"4ebd8142c891b5bcd532ec06f76c90fb115678295d1a5a87eee47442bb497dcb"} Oct 31 01:00:18.380651 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.380255 4969 scope.go:122] "RemoveContainer" containerID="8498623e6e349fa017fa19640c449cc4bf455ef5a6e4a7ab4c7e192fd916f856" Oct 31 01:00:18.380901 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.380672 4969 scope.go:122] "RemoveContainer" containerID="4ebd8142c891b5bcd532ec06f76c90fb115678295d1a5a87eee47442bb497dcb" Oct 31 01:00:18.381064 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:18.381044 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv_odh-ai-gateway-infra(e4813544-394e-4440-af14-a7b2a3c520be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" podUID="e4813544-394e-4440-af14-a7b2a3c520be" Oct 31 01:00:18.382377 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.382356 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" event={"ID":"d58a02d2-f107-4ce3-a2e8-680eb5583abc","Type":"ContainerStarted","Data":"0bad53903b157bb5ba0fdbcdd3b6dddb7aac5f57a496ad81eeba29f1dbec0436"} Oct 31 01:00:18.385990 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.385963 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" event={"ID":"2ea7dc39-a164-44d7-8959-ab89867c8100","Type":"ContainerStarted","Data":"fe34de29fe0fe6e8ae97513d9005a25f9539103446ef6d3794bd710156f3bbe2"} Oct 31 01:00:18.584749 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.584710 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:18.628556 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.628530 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:18.701893 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.701856 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2d406a-111c-4af4-95ba-b7e0304c399f-kube-api-access-kggrt\" (UniqueName: \"kubernetes.io/projected/ed2d406a-111c-4af4-95ba-b7e0304c399f-kube-api-access-kggrt\") pod \"ed2d406a-111c-4af4-95ba-b7e0304c399f\" (UID: \"ed2d406a-111c-4af4-95ba-b7e0304c399f\") " Oct 31 01:00:18.702046 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.701942 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed2d406a-111c-4af4-95ba-b7e0304c399f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d406a-111c-4af4-95ba-b7e0304c399f-maas-api-service-ca\") pod \"ed2d406a-111c-4af4-95ba-b7e0304c399f\" (UID: \"ed2d406a-111c-4af4-95ba-b7e0304c399f\") " Oct 31 01:00:18.702262 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.702241 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2d406a-111c-4af4-95ba-b7e0304c399f-maas-api-service-ca" pod "ed2d406a-111c-4af4-95ba-b7e0304c399f" (UID: "ed2d406a-111c-4af4-95ba-b7e0304c399f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:18.702442 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.702426 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d406a-111c-4af4-95ba-b7e0304c399f-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.738936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.738882 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv"] Oct 31 01:00:18.777546 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.777151 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2d406a-111c-4af4-95ba-b7e0304c399f-kube-api-access-kggrt" pod "ed2d406a-111c-4af4-95ba-b7e0304c399f" (UID: "ed2d406a-111c-4af4-95ba-b7e0304c399f"). InnerVolumeSpecName "kube-api-access-kggrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:18.803094 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.803065 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34afbdf-0019-4945-ae98-076e3faab54c-kube-api-access-d5wm4\" (UniqueName: \"kubernetes.io/projected/c34afbdf-0019-4945-ae98-076e3faab54c-kube-api-access-d5wm4\") pod \"c34afbdf-0019-4945-ae98-076e3faab54c\" (UID: \"c34afbdf-0019-4945-ae98-076e3faab54c\") " Oct 31 01:00:18.803248 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.803192 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34afbdf-0019-4945-ae98-076e3faab54c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34afbdf-0019-4945-ae98-076e3faab54c-maas-api-service-ca\") pod \"c34afbdf-0019-4945-ae98-076e3faab54c\" (UID: \"c34afbdf-0019-4945-ae98-076e3faab54c\") " Oct 31 01:00:18.803498 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.803479 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kggrt\" (UniqueName: \"kubernetes.io/projected/ed2d406a-111c-4af4-95ba-b7e0304c399f-kube-api-access-kggrt\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.803636 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.803505 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34afbdf-0019-4945-ae98-076e3faab54c-maas-api-service-ca" pod "c34afbdf-0019-4945-ae98-076e3faab54c" (UID: "c34afbdf-0019-4945-ae98-076e3faab54c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:18.805272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.805219 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34afbdf-0019-4945-ae98-076e3faab54c-kube-api-access-d5wm4" pod "c34afbdf-0019-4945-ae98-076e3faab54c" (UID: "c34afbdf-0019-4945-ae98-076e3faab54c"). InnerVolumeSpecName "kube-api-access-d5wm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:18.905334 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.905298 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wm4\" (UniqueName: \"kubernetes.io/projected/c34afbdf-0019-4945-ae98-076e3faab54c-kube-api-access-d5wm4\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.905334 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:18.905332 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34afbdf-0019-4945-ae98-076e3faab54c-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.396726 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.396697 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" Oct 31 01:00:19.397198 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.396733 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng" event={"ID":"c34afbdf-0019-4945-ae98-076e3faab54c","Type":"ContainerDied","Data":"9f21a8eb01e9e8e71d58cbeac510966a5b9db9832ca0a4adbca7e5da784818eb"} Oct 31 01:00:19.397198 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.396806 4969 scope.go:122] "RemoveContainer" containerID="5573f11883b78748e61e785a0aa1194561c1fee7b1e72b1cab64fa303cd02800" Oct 31 01:00:19.401444 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.401414 4969 generic.go:356] "Generic (PLEG): container finished" podID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerID="0bad53903b157bb5ba0fdbcdd3b6dddb7aac5f57a496ad81eeba29f1dbec0436" exitCode=60 Oct 31 01:00:19.401569 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.401482 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" event={"ID":"d58a02d2-f107-4ce3-a2e8-680eb5583abc","Type":"ContainerDied","Data":"0bad53903b157bb5ba0fdbcdd3b6dddb7aac5f57a496ad81eeba29f1dbec0436"} Oct 31 01:00:19.401974 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.401952 4969 scope.go:122] "RemoveContainer" containerID="0bad53903b157bb5ba0fdbcdd3b6dddb7aac5f57a496ad81eeba29f1dbec0436" Oct 31 01:00:19.402681 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:19.402313 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7_odh-ai-gateway-infra(d58a02d2-f107-4ce3-a2e8-680eb5583abc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" Oct 31 01:00:19.404225 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.403634 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" Oct 31 01:00:19.404225 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.403693 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv" event={"ID":"ed2d406a-111c-4af4-95ba-b7e0304c399f","Type":"ContainerDied","Data":"ec3dd26456d0735c78a6928d14a184effcdcd7d83830132d603ae992eae6b296"} Oct 31 01:00:19.423070 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.423027 4969 generic.go:356] "Generic (PLEG): container finished" podID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerID="fe34de29fe0fe6e8ae97513d9005a25f9539103446ef6d3794bd710156f3bbe2" exitCode=60 Oct 31 01:00:19.423184 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.423144 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" event={"ID":"2ea7dc39-a164-44d7-8959-ab89867c8100","Type":"ContainerDied","Data":"fe34de29fe0fe6e8ae97513d9005a25f9539103446ef6d3794bd710156f3bbe2"} Oct 31 01:00:19.423642 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.423620 4969 scope.go:122] "RemoveContainer" containerID="474cc21a18b43adfe1fa3a49b1efc816beffeed3ced4c5263ce6ed721afba438" Oct 31 01:00:19.424990 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.424951 4969 scope.go:122] "RemoveContainer" containerID="fe34de29fe0fe6e8ae97513d9005a25f9539103446ef6d3794bd710156f3bbe2" Oct 31 01:00:19.425567 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:19.425533 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-mmpsj_odh-ai-gateway-infra(2ea7dc39-a164-44d7-8959-ab89867c8100)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" Oct 31 01:00:19.451369 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.451341 4969 scope.go:122] "RemoveContainer" containerID="f50d2205e077c9cebe783ac227ad13998630ebe31a32d7dfb9cfde11af60e4ac" Oct 31 01:00:19.462704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.462666 4969 scope.go:122] "RemoveContainer" containerID="4d374d4c5259b75d27c3113d854d53d149f6c180a712483d676a362495223f8e" Oct 31 01:00:19.550123 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.550096 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:19.563633 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.563607 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng"] Oct 31 01:00:19.569765 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.569741 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a7eb09-29890140-j5kng"] Oct 31 01:00:19.716439 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.716352 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4813544-394e-4440-af14-a7b2a3c520be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4813544-394e-4440-af14-a7b2a3c520be-maas-api-service-ca\") pod \"e4813544-394e-4440-af14-a7b2a3c520be\" (UID: \"e4813544-394e-4440-af14-a7b2a3c520be\") " Oct 31 01:00:19.716790 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.716767 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4813544-394e-4440-af14-a7b2a3c520be-kube-api-access-tvlxp\" (UniqueName: \"kubernetes.io/projected/e4813544-394e-4440-af14-a7b2a3c520be-kube-api-access-tvlxp\") pod \"e4813544-394e-4440-af14-a7b2a3c520be\" (UID: \"e4813544-394e-4440-af14-a7b2a3c520be\") " Oct 31 01:00:19.717315 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.717082 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4813544-394e-4440-af14-a7b2a3c520be-maas-api-service-ca" pod "e4813544-394e-4440-af14-a7b2a3c520be" (UID: "e4813544-394e-4440-af14-a7b2a3c520be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.718171 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.718133 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4813544-394e-4440-af14-a7b2a3c520be-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.735272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.735232 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4813544-394e-4440-af14-a7b2a3c520be-kube-api-access-tvlxp" pod "e4813544-394e-4440-af14-a7b2a3c520be" (UID: "e4813544-394e-4440-af14-a7b2a3c520be"). InnerVolumeSpecName "kube-api-access-tvlxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.740329 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.740296 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" path="/var/lib/kubelet/pods/c34afbdf-0019-4945-ae98-076e3faab54c/volumes" Oct 31 01:00:19.744886 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.744818 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj"] Oct 31 01:00:19.812974 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.812915 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv"] Oct 31 01:00:19.817556 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.817529 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0144bd-29890140-w2rcv"] Oct 31 01:00:19.819842 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.819815 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlxp\" (UniqueName: \"kubernetes.io/projected/e4813544-394e-4440-af14-a7b2a3c520be-kube-api-access-tvlxp\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.972636 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:19.972547 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7"] Oct 31 01:00:20.435772 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.435737 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" Oct 31 01:00:20.435772 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.435767 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv" event={"ID":"e4813544-394e-4440-af14-a7b2a3c520be","Type":"ContainerDied","Data":"4f66e8d6847d6430612cda2b7422d190ccdc8599fe0eb730b491017b4a57af76"} Oct 31 01:00:20.436274 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.435835 4969 scope.go:122] "RemoveContainer" containerID="4ebd8142c891b5bcd532ec06f76c90fb115678295d1a5a87eee47442bb497dcb" Oct 31 01:00:20.642140 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.642115 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:20.644742 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.644721 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:20.731936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.731856 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58a02d2-f107-4ce3-a2e8-680eb5583abc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58a02d2-f107-4ce3-a2e8-680eb5583abc-maas-api-service-ca\") pod \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\" (UID: \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\") " Oct 31 01:00:20.732098 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.732030 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58a02d2-f107-4ce3-a2e8-680eb5583abc-kube-api-access-9qc8q\" (UniqueName: \"kubernetes.io/projected/d58a02d2-f107-4ce3-a2e8-680eb5583abc-kube-api-access-9qc8q\") pod \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\" (UID: \"d58a02d2-f107-4ce3-a2e8-680eb5583abc\") " Oct 31 01:00:20.732282 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.732259 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58a02d2-f107-4ce3-a2e8-680eb5583abc-maas-api-service-ca" pod "d58a02d2-f107-4ce3-a2e8-680eb5583abc" (UID: "d58a02d2-f107-4ce3-a2e8-680eb5583abc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:20.732559 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.732542 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58a02d2-f107-4ce3-a2e8-680eb5583abc-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:20.734349 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.734323 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58a02d2-f107-4ce3-a2e8-680eb5583abc-kube-api-access-9qc8q" pod "d58a02d2-f107-4ce3-a2e8-680eb5583abc" (UID: "d58a02d2-f107-4ce3-a2e8-680eb5583abc"). InnerVolumeSpecName "kube-api-access-9qc8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:20.834048 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.834018 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea7dc39-a164-44d7-8959-ab89867c8100-kube-api-access-k7jhf\" (UniqueName: \"kubernetes.io/projected/2ea7dc39-a164-44d7-8959-ab89867c8100-kube-api-access-k7jhf\") pod \"2ea7dc39-a164-44d7-8959-ab89867c8100\" (UID: \"2ea7dc39-a164-44d7-8959-ab89867c8100\") " Oct 31 01:00:20.834244 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.834097 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea7dc39-a164-44d7-8959-ab89867c8100-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea7dc39-a164-44d7-8959-ab89867c8100-maas-api-service-ca\") pod \"2ea7dc39-a164-44d7-8959-ab89867c8100\" (UID: \"2ea7dc39-a164-44d7-8959-ab89867c8100\") " Oct 31 01:00:20.834458 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.834441 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qc8q\" (UniqueName: \"kubernetes.io/projected/d58a02d2-f107-4ce3-a2e8-680eb5583abc-kube-api-access-9qc8q\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:20.834551 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.834517 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea7dc39-a164-44d7-8959-ab89867c8100-maas-api-service-ca" pod "2ea7dc39-a164-44d7-8959-ab89867c8100" (UID: "2ea7dc39-a164-44d7-8959-ab89867c8100"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:20.836441 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.836417 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea7dc39-a164-44d7-8959-ab89867c8100-kube-api-access-k7jhf" pod "2ea7dc39-a164-44d7-8959-ab89867c8100" (UID: "2ea7dc39-a164-44d7-8959-ab89867c8100"). InnerVolumeSpecName "kube-api-access-k7jhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:20.850445 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.850418 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv"] Oct 31 01:00:20.861002 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.860975 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f13553-29890140-2chrv"] Oct 31 01:00:20.935984 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.935954 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea7dc39-a164-44d7-8959-ab89867c8100-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:20.935984 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:20.935981 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7jhf\" (UniqueName: \"kubernetes.io/projected/2ea7dc39-a164-44d7-8959-ab89867c8100-kube-api-access-k7jhf\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:00:21.449967 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.449940 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" Oct 31 01:00:21.450495 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.449939 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj" event={"ID":"2ea7dc39-a164-44d7-8959-ab89867c8100","Type":"ContainerDied","Data":"cff064ee445069769a3ae94102780523c01e74a9b0ab995897903781e8360a42"} Oct 31 01:00:21.450495 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.450077 4969 scope.go:122] "RemoveContainer" containerID="fe34de29fe0fe6e8ae97513d9005a25f9539103446ef6d3794bd710156f3bbe2" Oct 31 01:00:21.452967 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.452949 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" Oct 31 01:00:21.452967 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.452953 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7" event={"ID":"d58a02d2-f107-4ce3-a2e8-680eb5583abc","Type":"ContainerDied","Data":"a5ed520b79a92a857f93b84b38bb7a4ae7c9e6f9075c7b037039cdc6edfdf23a"} Oct 31 01:00:21.462504 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.462485 4969 scope.go:122] "RemoveContainer" containerID="0bad53903b157bb5ba0fdbcdd3b6dddb7aac5f57a496ad81eeba29f1dbec0436" Oct 31 01:00:21.615098 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.615056 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7"] Oct 31 01:00:21.664384 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.664351 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfc055-29890140-54vp7"] Oct 31 01:00:21.735861 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.735786 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" path="/var/lib/kubelet/pods/d58a02d2-f107-4ce3-a2e8-680eb5583abc/volumes" Oct 31 01:00:21.736268 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.736253 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4813544-394e-4440-af14-a7b2a3c520be" path="/var/lib/kubelet/pods/e4813544-394e-4440-af14-a7b2a3c520be/volumes" Oct 31 01:00:21.736678 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.736664 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" path="/var/lib/kubelet/pods/ed2d406a-111c-4af4-95ba-b7e0304c399f/volumes" Oct 31 01:00:21.776208 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.776173 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj"] Oct 31 01:00:21.781453 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:21.781429 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-mmpsj"] Oct 31 01:00:23.735504 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:23.735467 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" path="/var/lib/kubelet/pods/2ea7dc39-a164-44d7-8959-ab89867c8100/volumes" Oct 31 01:00:32.731794 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:00:32.731756 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:00:34.532284 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.532241 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674"] Oct 31 01:00:34.533031 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533007 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerName="cleanup" Oct 31 01:00:34.533031 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533032 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533047 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e4813544-394e-4440-af14-a7b2a3c520be" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533055 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4813544-394e-4440-af14-a7b2a3c520be" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533078 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533086 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533101 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533109 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533121 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e4813544-394e-4440-af14-a7b2a3c520be" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533128 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4813544-394e-4440-af14-a7b2a3c520be" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533145 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533152 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533164 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533171 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533187 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533193 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533209 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" containerName="cleanup" Oct 31 01:00:34.533207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533216 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533227 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e4813544-394e-4440-af14-a7b2a3c520be" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533233 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4813544-394e-4440-af14-a7b2a3c520be" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533261 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533267 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533288 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533294 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533432 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533446 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4813544-394e-4440-af14-a7b2a3c520be" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533457 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533466 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533474 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4813544-394e-4440-af14-a7b2a3c520be" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533484 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533494 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533503 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533512 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34afbdf-0019-4945-ae98-076e3faab54c" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533522 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4813544-394e-4440-af14-a7b2a3c520be" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533532 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533542 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533550 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerName="cleanup" Oct 31 01:00:34.534464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.533559 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerName="cleanup" Oct 31 01:00:34.539449 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.539425 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:00:34.600585 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.600542 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dhx\" (UniqueName: \"kubernetes.io/projected/36e09f55-204e-430c-8cc7-47fb2356889a-kube-api-access-f2dhx\") pod \"payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674\" (UID: \"36e09f55-204e-430c-8cc7-47fb2356889a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:00:34.600819 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.600737 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e09f55-204e-430c-8cc7-47fb2356889a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674\" (UID: \"36e09f55-204e-430c-8cc7-47fb2356889a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:00:34.601558 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.601532 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674"] Oct 31 01:00:34.639736 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.639706 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e8808d\"" Oct 31 01:00:34.642325 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.642304 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e8808d-dockercfg-r7xm6\"" Oct 31 01:00:34.702389 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.702354 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e09f55-204e-430c-8cc7-47fb2356889a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674\" (UID: \"36e09f55-204e-430c-8cc7-47fb2356889a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:00:34.702537 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.702494 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dhx\" (UniqueName: \"kubernetes.io/projected/36e09f55-204e-430c-8cc7-47fb2356889a-kube-api-access-f2dhx\") pod \"payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674\" (UID: \"36e09f55-204e-430c-8cc7-47fb2356889a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:00:34.703066 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.703045 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e09f55-204e-430c-8cc7-47fb2356889a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674\" (UID: \"36e09f55-204e-430c-8cc7-47fb2356889a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:00:34.763567 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.763534 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dhx\" (UniqueName: \"kubernetes.io/projected/36e09f55-204e-430c-8cc7-47fb2356889a-kube-api-access-f2dhx\") pod \"payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674\" (UID: \"36e09f55-204e-430c-8cc7-47fb2356889a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:00:34.876824 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:34.876799 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:00:35.266250 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:35.266221 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674"] Oct 31 01:00:35.267204 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:00:35.267178 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e09f55_204e_430c_8cc7_47fb2356889a.slice/crio-a382245431d999d95e5ed1fb8f160886f0f431d06cb79a3154f000de15004bcc WatchSource:0}: Error finding container a382245431d999d95e5ed1fb8f160886f0f431d06cb79a3154f000de15004bcc: Status 404 returned error can't find the container with id a382245431d999d95e5ed1fb8f160886f0f431d06cb79a3154f000de15004bcc Oct 31 01:00:35.592272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:35.592176 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" event={"ID":"36e09f55-204e-430c-8cc7-47fb2356889a","Type":"ContainerStarted","Data":"37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9"} Oct 31 01:00:35.592272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:35.592241 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" event={"ID":"36e09f55-204e-430c-8cc7-47fb2356889a","Type":"ContainerStarted","Data":"a382245431d999d95e5ed1fb8f160886f0f431d06cb79a3154f000de15004bcc"} Oct 31 01:00:35.592762 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:35.592368 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:00:35.711452 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:35.711399 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" podStartSLOduration=1.7113861099999998 podStartE2EDuration="1.71138611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:35.708093579 +0000 UTC m=+3468.183452003" watchObservedRunningTime="2026-10-31 01:00:35.71138611 +0000 UTC m=+3468.186744535" Oct 31 01:00:46.602785 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:00:46.602753 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:01:02.727846 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:01:02.727808 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:01:32.729456 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:01:32.729419 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:01:46.072676 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:46.072640 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w"] Oct 31 01:01:58.535167 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.535089 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs"] Oct 31 01:01:58.535784 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.535767 4969 cpu_manager.go:414] "containerMap: removing container" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerName="cleanup" Oct 31 01:01:58.535836 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.535785 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerName="cleanup" Oct 31 01:01:58.535836 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.535804 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerName="cleanup" Oct 31 01:01:58.535836 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.535809 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea7dc39-a164-44d7-8959-ab89867c8100" containerName="cleanup" Oct 31 01:01:58.535836 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.535829 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerName="cleanup" Oct 31 01:01:58.535836 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.535834 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2d406a-111c-4af4-95ba-b7e0304c399f" containerName="cleanup" Oct 31 01:01:58.536003 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.535933 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58a02d2-f107-4ce3-a2e8-680eb5583abc" containerName="cleanup" Oct 31 01:01:58.539696 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.539677 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.593574 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.593550 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-aaa842-serving-cert\"" Oct 31 01:01:58.647289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.647256 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs"] Oct 31 01:01:58.659375 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.659344 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.659624 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.659573 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4234bd78-5607-40a7-8f67-94848b2cf4be-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.659710 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.659643 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-metrics-certs\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.659773 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.659708 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8gx\" (UniqueName: \"kubernetes.io/projected/4234bd78-5607-40a7-8f67-94848b2cf4be-kube-api-access-6h8gx\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.760697 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.760576 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4234bd78-5607-40a7-8f67-94848b2cf4be-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.760882 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.760740 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-metrics-certs\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.760882 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.760782 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8gx\" (UniqueName: \"kubernetes.io/projected/4234bd78-5607-40a7-8f67-94848b2cf4be-kube-api-access-6h8gx\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.761006 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.760882 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.761063 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:01:58.761042 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-serving-cert: secret "maas-api-e2e-extmod-a-aaa842-serving-cert" not found Oct 31 01:01:58.761145 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:01:58.761129 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls podName:4234bd78-5607-40a7-8f67-94848b2cf4be nodeName:}" failed. No retries permitted until 2026-10-31 01:01:59.261107662 +0000 UTC m=+3551.736466064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls") pod "maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" (UID: "4234bd78-5607-40a7-8f67-94848b2cf4be") : secret "maas-api-e2e-extmod-a-aaa842-serving-cert" not found Oct 31 01:01:58.761399 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.761372 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4234bd78-5607-40a7-8f67-94848b2cf4be-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.763422 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.763404 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-metrics-certs\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:58.824852 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:58.824777 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8gx\" (UniqueName: \"kubernetes.io/projected/4234bd78-5607-40a7-8f67-94848b2cf4be-kube-api-access-6h8gx\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:59.267070 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:59.267030 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:59.269852 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:59.269825 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:59.454235 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:59.454202 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:01:59.834712 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:01:59.834676 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs"] Oct 31 01:01:59.835157 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:01:59.834883 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4234bd78_5607_40a7_8f67_94848b2cf4be.slice/crio-e8779d6196fe7eab2bdab9ebf72ce8dc26133c6de3d4cb14e46c378167695602 WatchSource:0}: Error finding container e8779d6196fe7eab2bdab9ebf72ce8dc26133c6de3d4cb14e46c378167695602: Status 404 returned error can't find the container with id e8779d6196fe7eab2bdab9ebf72ce8dc26133c6de3d4cb14e46c378167695602 Oct 31 01:02:00.259561 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:00.259527 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" event={"ID":"4234bd78-5607-40a7-8f67-94848b2cf4be","Type":"ContainerStarted","Data":"e8779d6196fe7eab2bdab9ebf72ce8dc26133c6de3d4cb14e46c378167695602"} Oct 31 01:02:01.093982 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.093889 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" podUID="4796847a-ea8b-4fc8-8c21-8b7d145866f8" containerName="main" containerID="cri-o://d4710bbe6947be3f759b68c17708c04d506036b53dd4d5cbd04ec9a969b2cf86" gracePeriod=45 Oct 31 01:02:01.268185 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.268154 4969 generic.go:356] "Generic (PLEG): container finished" podID="4796847a-ea8b-4fc8-8c21-8b7d145866f8" containerID="d4710bbe6947be3f759b68c17708c04d506036b53dd4d5cbd04ec9a969b2cf86" exitCode=0 Oct 31 01:02:01.268372 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.268228 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" event={"ID":"4796847a-ea8b-4fc8-8c21-8b7d145866f8","Type":"ContainerDied","Data":"d4710bbe6947be3f759b68c17708c04d506036b53dd4d5cbd04ec9a969b2cf86"} Oct 31 01:02:01.270015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.269984 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" event={"ID":"4234bd78-5607-40a7-8f67-94848b2cf4be","Type":"ContainerStarted","Data":"5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325"} Oct 31 01:02:01.270171 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.270092 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:02:01.376290 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.376261 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 01:02:01.391168 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.391114 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" podStartSLOduration=2.537951588 podStartE2EDuration="3.391099964s" totalImagesPullingTime="853.148376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:01.387407687 +0000 UTC m=+3553.862766113" watchObservedRunningTime="2026-10-31 01:02:01.391099964 +0000 UTC m=+3553.866458433" Oct 31 01:02:01.494663 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.494631 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-home\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-home\") pod \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " Oct 31 01:02:01.494863 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.494740 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-model-cache\") pod \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " Oct 31 01:02:01.494863 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.494789 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4796847a-ea8b-4fc8-8c21-8b7d145866f8-kube-api-access-77vp7\" (UniqueName: \"kubernetes.io/projected/4796847a-ea8b-4fc8-8c21-8b7d145866f8-kube-api-access-77vp7\") pod \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " Oct 31 01:02:01.494863 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.494852 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tls-certs\") pod \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " Oct 31 01:02:01.495054 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.494915 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-dshm\") pod \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " Oct 31 01:02:01.495054 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.494916 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-home" pod "4796847a-ea8b-4fc8-8c21-8b7d145866f8" (UID: "4796847a-ea8b-4fc8-8c21-8b7d145866f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:01.495054 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.494943 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tmp-dir\") pod \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\" (UID: \"4796847a-ea8b-4fc8-8c21-8b7d145866f8\") " Oct 31 01:02:01.495054 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.495032 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-model-cache" pod "4796847a-ea8b-4fc8-8c21-8b7d145866f8" (UID: "4796847a-ea8b-4fc8-8c21-8b7d145866f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:01.495290 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.495259 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tmp-dir" pod "4796847a-ea8b-4fc8-8c21-8b7d145866f8" (UID: "4796847a-ea8b-4fc8-8c21-8b7d145866f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:01.495578 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.495530 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:01.495578 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.495555 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:01.495578 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.495568 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:01.497504 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.497477 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tls-certs" pod "4796847a-ea8b-4fc8-8c21-8b7d145866f8" (UID: "4796847a-ea8b-4fc8-8c21-8b7d145866f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:01.497610 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.497485 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-dshm" pod "4796847a-ea8b-4fc8-8c21-8b7d145866f8" (UID: "4796847a-ea8b-4fc8-8c21-8b7d145866f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:01.497610 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.497485 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4796847a-ea8b-4fc8-8c21-8b7d145866f8-kube-api-access-77vp7" pod "4796847a-ea8b-4fc8-8c21-8b7d145866f8" (UID: "4796847a-ea8b-4fc8-8c21-8b7d145866f8"). InnerVolumeSpecName "kube-api-access-77vp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:01.597389 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.597344 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4796847a-ea8b-4fc8-8c21-8b7d145866f8-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:01.597389 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.597381 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4796847a-ea8b-4fc8-8c21-8b7d145866f8-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:01.597642 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:01.597393 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77vp7\" (UniqueName: \"kubernetes.io/projected/4796847a-ea8b-4fc8-8c21-8b7d145866f8-kube-api-access-77vp7\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:02.278841 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:02.278807 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" Oct 31 01:02:02.278841 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:02.278817 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w" event={"ID":"4796847a-ea8b-4fc8-8c21-8b7d145866f8","Type":"ContainerDied","Data":"d83c8d15b82bc841e173172d67661d2872e615afa28218ffd2b78fcfca71ffbf"} Oct 31 01:02:02.279414 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:02.278876 4969 scope.go:122] "RemoveContainer" containerID="d4710bbe6947be3f759b68c17708c04d506036b53dd4d5cbd04ec9a969b2cf86" Oct 31 01:02:02.445212 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:02.445172 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w"] Oct 31 01:02:02.494695 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:02.494659 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e1a6c/auth-test-model-4e1a6c-kserve-5d8f6b748f-cfv6w"] Oct 31 01:02:02.728370 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:02:02.728338 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:02:03.736039 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:03.736007 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4796847a-ea8b-4fc8-8c21-8b7d145866f8" path="/var/lib/kubelet/pods/4796847a-ea8b-4fc8-8c21-8b7d145866f8/volumes" Oct 31 01:02:07.284113 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:07.284084 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:02:07.863210 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:07.863180 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws"] Oct 31 01:02:22.884491 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:22.884449 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" podUID="778e00fa-3daa-47b9-866e-4c742d50333b" containerName="main" containerID="cri-o://08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6" gracePeriod=45 Oct 31 01:02:23.181993 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.181966 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 01:02:23.271252 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.271218 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-model-cache\") pod \"778e00fa-3daa-47b9-866e-4c742d50333b\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " Oct 31 01:02:23.271435 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.271327 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778e00fa-3daa-47b9-866e-4c742d50333b-kube-api-access-nbjrg\" (UniqueName: \"kubernetes.io/projected/778e00fa-3daa-47b9-866e-4c742d50333b-kube-api-access-nbjrg\") pod \"778e00fa-3daa-47b9-866e-4c742d50333b\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " Oct 31 01:02:23.271435 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.271353 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-dshm\") pod \"778e00fa-3daa-47b9-866e-4c742d50333b\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " Oct 31 01:02:23.271435 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.271418 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/778e00fa-3daa-47b9-866e-4c742d50333b-tls-certs\" (UniqueName: \"kubernetes.io/secret/778e00fa-3daa-47b9-866e-4c742d50333b-tls-certs\") pod \"778e00fa-3daa-47b9-866e-4c742d50333b\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " Oct 31 01:02:23.271680 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.271476 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-model-cache" pod "778e00fa-3daa-47b9-866e-4c742d50333b" (UID: "778e00fa-3daa-47b9-866e-4c742d50333b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:23.271680 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.271514 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-home\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-home\") pod \"778e00fa-3daa-47b9-866e-4c742d50333b\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " Oct 31 01:02:23.271680 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.271659 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-tmp-dir\") pod \"778e00fa-3daa-47b9-866e-4c742d50333b\" (UID: \"778e00fa-3daa-47b9-866e-4c742d50333b\") " Oct 31 01:02:23.271872 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.271785 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-home" pod "778e00fa-3daa-47b9-866e-4c742d50333b" (UID: "778e00fa-3daa-47b9-866e-4c742d50333b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:23.271936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.271904 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-tmp-dir" pod "778e00fa-3daa-47b9-866e-4c742d50333b" (UID: "778e00fa-3daa-47b9-866e-4c742d50333b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:23.272422 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.272398 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:23.272533 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.272427 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:23.272533 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.272445 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:23.273990 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.273963 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-dshm" pod "778e00fa-3daa-47b9-866e-4c742d50333b" (UID: "778e00fa-3daa-47b9-866e-4c742d50333b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:23.274073 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.273990 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778e00fa-3daa-47b9-866e-4c742d50333b-tls-certs" pod "778e00fa-3daa-47b9-866e-4c742d50333b" (UID: "778e00fa-3daa-47b9-866e-4c742d50333b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:23.274073 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.274058 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778e00fa-3daa-47b9-866e-4c742d50333b-kube-api-access-nbjrg" pod "778e00fa-3daa-47b9-866e-4c742d50333b" (UID: "778e00fa-3daa-47b9-866e-4c742d50333b"). InnerVolumeSpecName "kube-api-access-nbjrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:23.373961 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.373922 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778e00fa-3daa-47b9-866e-4c742d50333b-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:23.373961 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.373956 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbjrg\" (UniqueName: \"kubernetes.io/projected/778e00fa-3daa-47b9-866e-4c742d50333b-kube-api-access-nbjrg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:23.374160 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.373972 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778e00fa-3daa-47b9-866e-4c742d50333b-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:02:23.438667 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.438553 4969 generic.go:356] "Generic (PLEG): container finished" podID="778e00fa-3daa-47b9-866e-4c742d50333b" containerID="08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6" exitCode=0 Oct 31 01:02:23.438667 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.438619 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" event={"ID":"778e00fa-3daa-47b9-866e-4c742d50333b","Type":"ContainerDied","Data":"08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6"} Oct 31 01:02:23.438914 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.438682 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" event={"ID":"778e00fa-3daa-47b9-866e-4c742d50333b","Type":"ContainerDied","Data":"c285fc977fdb4f5d658ff91cc440d169ada1ed13b1fbb7d5ff09a9be8b185a0f"} Oct 31 01:02:23.438914 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.438705 4969 scope.go:122] "RemoveContainer" containerID="08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6" Oct 31 01:02:23.438914 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.438634 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws" Oct 31 01:02:23.450700 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.450681 4969 scope.go:122] "RemoveContainer" containerID="08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6" Oct 31 01:02:23.450964 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:02:23.450943 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6\": container with ID starting with 08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6 not found: ID does not exist" containerID="08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6" Oct 31 01:02:23.451091 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.450978 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6"} err="failed to get container status \"08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6\": rpc error: code = NotFound desc = could not find container \"08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6\": container with ID starting with 08f68eaf9269d8e0be6369917b783b29781824f4dba4dad32830bc27dc0811a6 not found: ID does not exist" Oct 31 01:02:23.604513 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.604478 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws"] Oct 31 01:02:23.608326 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.608299 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4743f8/auth-test-model-4743f8-kserve-56f459d485-866ws"] Oct 31 01:02:23.735494 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:23.735421 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778e00fa-3daa-47b9-866e-4c742d50333b" path="/var/lib/kubelet/pods/778e00fa-3daa-47b9-866e-4c742d50333b/volumes" Oct 31 01:02:32.728665 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:02:32.728632 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:02:39.660217 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.660179 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg"] Oct 31 01:02:39.660949 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.660933 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4796847a-ea8b-4fc8-8c21-8b7d145866f8" containerName="main" Oct 31 01:02:39.661002 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.660951 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4796847a-ea8b-4fc8-8c21-8b7d145866f8" containerName="main" Oct 31 01:02:39.661002 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.660972 4969 cpu_manager.go:414] "containerMap: removing container" podUID="778e00fa-3daa-47b9-866e-4c742d50333b" containerName="main" Oct 31 01:02:39.661002 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.660977 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778e00fa-3daa-47b9-866e-4c742d50333b" containerName="main" Oct 31 01:02:39.661099 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.661086 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4796847a-ea8b-4fc8-8c21-8b7d145866f8" containerName="main" Oct 31 01:02:39.661099 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.661095 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="778e00fa-3daa-47b9-866e-4c742d50333b" containerName="main" Oct 31 01:02:39.665927 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.665910 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.701109 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.701063 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-socket\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.701297 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.701147 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-token\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.701297 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.701208 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-envoy\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.701297 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.701246 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-certs\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.701297 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.701282 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndf8\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-kube-api-access-mndf8\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.701832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.701358 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-data\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.701832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.701498 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-credential-socket\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.701832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.701675 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ca7f1-d050-4b15-917b-e901168b2199-istiod-ca-cert\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.701832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.701747 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-podinfo\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.726708 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.726672 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2aebbd-openshift-default-dockercfg-rxhnf\"" Oct 31 01:02:39.781653 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.781618 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg"] Oct 31 01:02:39.802912 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.802875 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-socket\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.803290 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.803256 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-token\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.803452 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.803341 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-envoy\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.803452 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.803382 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-certs\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.803452 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.803420 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndf8\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-kube-api-access-mndf8\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.803724 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.803450 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-data\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.803724 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.803483 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-credential-socket\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.803936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.803910 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-certs\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.804497 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.804472 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-data\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.804696 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.803336 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-socket\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.805160 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.805139 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-credential-socket\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.818498 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.812832 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ca7f1-d050-4b15-917b-e901168b2199-istiod-ca-cert\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.818498 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.812938 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-podinfo\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.818498 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.814375 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ca7f1-d050-4b15-917b-e901168b2199-istiod-ca-cert\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.828800 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.828721 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-envoy\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.834837 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.832701 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-podinfo\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.864458 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.864423 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-token\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.864669 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.864647 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndf8\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-kube-api-access-mndf8\") pod \"e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:39.981418 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:39.981327 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:40.416176 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:40.416151 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg"] Oct 31 01:02:40.416302 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:02:40.416228 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2ca7f1_d050_4b15_917b_e901168b2199.slice/crio-68c22127e092b0267fe68923d10187afbbe986bc1b3f2299734badf5c760ced6 WatchSource:0}: Error finding container 68c22127e092b0267fe68923d10187afbbe986bc1b3f2299734badf5c760ced6: Status 404 returned error can't find the container with id 68c22127e092b0267fe68923d10187afbbe986bc1b3f2299734badf5c760ced6 Oct 31 01:02:40.418976 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:40.418945 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:02:40.419059 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:40.419010 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:02:40.419059 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:40.419042 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:02:40.596137 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:40.596087 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" event={"ID":"cf2ca7f1-d050-4b15-917b-e901168b2199","Type":"ContainerStarted","Data":"15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472"} Oct 31 01:02:40.596343 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:40.596163 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" event={"ID":"cf2ca7f1-d050-4b15-917b-e901168b2199","Type":"ContainerStarted","Data":"68c22127e092b0267fe68923d10187afbbe986bc1b3f2299734badf5c760ced6"} Oct 31 01:02:40.745918 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:40.745807 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" podStartSLOduration=1.74579321 podStartE2EDuration="1.74579321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:40.742934391 +0000 UTC m=+3593.218292813" watchObservedRunningTime="2026-10-31 01:02:40.74579321 +0000 UTC m=+3593.221151632" Oct 31 01:02:40.981679 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:40.981649 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:41.988346 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:41.988315 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:42.610629 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:42.610584 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:42.611487 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:42.611465 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:02:45.144203 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.144171 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d"] Oct 31 01:02:45.148761 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.148737 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:02:45.253830 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.253790 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5498e8-dockercfg-ms75q\"" Oct 31 01:02:45.254040 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.253857 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5498e8\"" Oct 31 01:02:45.260990 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.260288 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d"] Oct 31 01:02:45.295480 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.295430 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcww8\" (UniqueName: \"kubernetes.io/projected/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-kube-api-access-vcww8\") pod \"payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d\" (UID: \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:02:45.295731 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.295503 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d\" (UID: \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:02:45.397289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.397211 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcww8\" (UniqueName: \"kubernetes.io/projected/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-kube-api-access-vcww8\") pod \"payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d\" (UID: \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:02:45.397289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.397267 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d\" (UID: \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:02:45.398065 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.398039 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d\" (UID: \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:02:45.460010 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.459976 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcww8\" (UniqueName: \"kubernetes.io/projected/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-kube-api-access-vcww8\") pod \"payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d\" (UID: \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:02:45.463860 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.463834 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:02:45.845778 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:45.845750 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d"] Oct 31 01:02:45.846703 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:02:45.846673 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a5be49_af4b_45cf_9f8d_8fc49eb4ba75.slice/crio-5fe3c26734aa74d4ce6e55ff9588ee12f88b14b6070625a109e32ad6e72e1606 WatchSource:0}: Error finding container 5fe3c26734aa74d4ce6e55ff9588ee12f88b14b6070625a109e32ad6e72e1606: Status 404 returned error can't find the container with id 5fe3c26734aa74d4ce6e55ff9588ee12f88b14b6070625a109e32ad6e72e1606 Oct 31 01:02:46.643797 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:46.643759 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" event={"ID":"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75","Type":"ContainerStarted","Data":"3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8"} Oct 31 01:02:46.644212 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:46.643819 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" event={"ID":"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75","Type":"ContainerStarted","Data":"5fe3c26734aa74d4ce6e55ff9588ee12f88b14b6070625a109e32ad6e72e1606"} Oct 31 01:02:46.644212 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:46.643850 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:02:46.773897 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:46.773839 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" podStartSLOduration=1.773822047 podStartE2EDuration="1.773822047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:46.771681138 +0000 UTC m=+3599.247039560" watchObservedRunningTime="2026-10-31 01:02:46.773822047 +0000 UTC m=+3599.249180469" Oct 31 01:02:48.384299 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:48.384267 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 01:02:48.400623 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:48.400563 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 01:02:48.413048 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:48.413027 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 01:02:48.428581 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:48.428562 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 01:02:48.818098 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:48.818009 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4"] Oct 31 01:02:57.654312 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:02:57.654283 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:03:02.731914 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:03:02.731876 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:03:03.839700 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:03.839664 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" podUID="60ad44f3-a7fe-4664-8ae5-538e73aec5fe" containerName="main" containerID="cri-o://335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32" gracePeriod=45 Oct 31 01:03:03.991822 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:03.991768 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" podUID="60ad44f3-a7fe-4664-8ae5-538e73aec5fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.232:8000/ready\": dial tcp 10.133.0.232:8000: connect: connection refused" Oct 31 01:03:04.108349 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.108288 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 01:03:04.163137 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163108 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-model-cache\") pod \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " Oct 31 01:03:04.163289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163186 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tmp-dir\") pod \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " Oct 31 01:03:04.163289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163210 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-home\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-home\") pod \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " Oct 31 01:03:04.163289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163249 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-kube-api-access-fmfsn\" (UniqueName: \"kubernetes.io/projected/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-kube-api-access-fmfsn\") pod \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " Oct 31 01:03:04.163289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163279 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-dshm\") pod \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " Oct 31 01:03:04.163532 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163413 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-model-cache" pod "60ad44f3-a7fe-4664-8ae5-538e73aec5fe" (UID: "60ad44f3-a7fe-4664-8ae5-538e73aec5fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:04.163532 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163425 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tls-certs\") pod \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\" (UID: \"60ad44f3-a7fe-4664-8ae5-538e73aec5fe\") " Oct 31 01:03:04.163532 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163453 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-home" pod "60ad44f3-a7fe-4664-8ae5-538e73aec5fe" (UID: "60ad44f3-a7fe-4664-8ae5-538e73aec5fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:04.163532 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163469 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tmp-dir" pod "60ad44f3-a7fe-4664-8ae5-538e73aec5fe" (UID: "60ad44f3-a7fe-4664-8ae5-538e73aec5fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:04.164006 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.163981 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:04.164006 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.164007 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:04.164006 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.164022 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:04.165754 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.165732 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-dshm" pod "60ad44f3-a7fe-4664-8ae5-538e73aec5fe" (UID: "60ad44f3-a7fe-4664-8ae5-538e73aec5fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:04.165861 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.165752 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tls-certs" pod "60ad44f3-a7fe-4664-8ae5-538e73aec5fe" (UID: "60ad44f3-a7fe-4664-8ae5-538e73aec5fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:04.165861 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.165785 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-kube-api-access-fmfsn" pod "60ad44f3-a7fe-4664-8ae5-538e73aec5fe" (UID: "60ad44f3-a7fe-4664-8ae5-538e73aec5fe"). InnerVolumeSpecName "kube-api-access-fmfsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:04.264826 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.264794 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmfsn\" (UniqueName: \"kubernetes.io/projected/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-kube-api-access-fmfsn\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:04.264826 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.264823 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:04.265040 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.264834 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ad44f3-a7fe-4664-8ae5-538e73aec5fe-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:04.783368 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.783328 4969 generic.go:356] "Generic (PLEG): container finished" podID="60ad44f3-a7fe-4664-8ae5-538e73aec5fe" containerID="335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32" exitCode=0 Oct 31 01:03:04.783562 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.783398 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" Oct 31 01:03:04.783562 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.783405 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" event={"ID":"60ad44f3-a7fe-4664-8ae5-538e73aec5fe","Type":"ContainerDied","Data":"335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32"} Oct 31 01:03:04.783562 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.783450 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4" event={"ID":"60ad44f3-a7fe-4664-8ae5-538e73aec5fe","Type":"ContainerDied","Data":"bc75102cce59034186d7a2e87e30faf6fbb0bf3eda8a83a780cf19dfe809b850"} Oct 31 01:03:04.783562 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.783467 4969 scope.go:122] "RemoveContainer" containerID="335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32" Oct 31 01:03:04.795955 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.795753 4969 scope.go:122] "RemoveContainer" containerID="335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32" Oct 31 01:03:04.796366 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:03:04.796339 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32\": container with ID starting with 335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32 not found: ID does not exist" containerID="335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32" Oct 31 01:03:04.796496 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.796375 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32"} err="failed to get container status \"335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32\": rpc error: code = NotFound desc = could not find container \"335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32\": container with ID starting with 335b65b45978ff36ec51059d58cff1033f44520898e8bb2b6b2903d6626fde32 not found: ID does not exist" Oct 31 01:03:04.950414 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.950383 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4"] Oct 31 01:03:04.999250 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:04.999220 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-bf325c/auth-test-model-bf325c-kserve-758bb96ccc-bg7j4"] Oct 31 01:03:05.735187 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:05.735152 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ad44f3-a7fe-4664-8ae5-538e73aec5fe" path="/var/lib/kubelet/pods/60ad44f3-a7fe-4664-8ae5-538e73aec5fe/volumes" Oct 31 01:03:17.105960 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.105926 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24"] Oct 31 01:03:17.106633 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.106617 4969 cpu_manager.go:414] "containerMap: removing container" podUID="60ad44f3-a7fe-4664-8ae5-538e73aec5fe" containerName="main" Oct 31 01:03:17.106695 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.106635 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ad44f3-a7fe-4664-8ae5-538e73aec5fe" containerName="main" Oct 31 01:03:17.106791 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.106782 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ad44f3-a7fe-4664-8ae5-538e73aec5fe" containerName="main" Oct 31 01:03:17.111644 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.111624 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.119638 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.119585 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-metrics-certs\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.119768 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.119749 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.119818 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.119785 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89b6e-0836-4a95-adfb-fb359827d28e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.119936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.119921 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnvb\" (UniqueName: \"kubernetes.io/projected/64e89b6e-0836-4a95-adfb-fb359827d28e-kube-api-access-6dnvb\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.164439 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.164411 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-cc6150-serving-cert\"" Oct 31 01:03:17.218023 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.217987 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24"] Oct 31 01:03:17.220762 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.220738 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnvb\" (UniqueName: \"kubernetes.io/projected/64e89b6e-0836-4a95-adfb-fb359827d28e-kube-api-access-6dnvb\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.220904 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.220805 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-metrics-certs\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.220904 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.220871 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.220904 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.220892 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89b6e-0836-4a95-adfb-fb359827d28e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.221043 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:03:17.220999 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-serving-cert: secret "maas-api-e2e-extmod-a-cc6150-serving-cert" not found Oct 31 01:03:17.221091 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:03:17.221078 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls podName:64e89b6e-0836-4a95-adfb-fb359827d28e nodeName:}" failed. No retries permitted until 2026-10-31 01:03:17.721054073 +0000 UTC m=+3630.196412488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls") pod "maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" (UID: "64e89b6e-0836-4a95-adfb-fb359827d28e") : secret "maas-api-e2e-extmod-a-cc6150-serving-cert" not found Oct 31 01:03:17.221509 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.221490 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89b6e-0836-4a95-adfb-fb359827d28e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.223661 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.223634 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-metrics-certs\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.283735 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.283700 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnvb\" (UniqueName: \"kubernetes.io/projected/64e89b6e-0836-4a95-adfb-fb359827d28e-kube-api-access-6dnvb\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.726469 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.726429 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:17.729231 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:17.729199 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:18.026114 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:18.026021 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:18.463038 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:18.463011 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24"] Oct 31 01:03:18.887119 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:18.887084 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" event={"ID":"64e89b6e-0836-4a95-adfb-fb359827d28e","Type":"ContainerStarted","Data":"9d4fe0ea6434e22e2c3a1645deb6aae1ad597290644f862f8f81a75c9118448f"} Oct 31 01:03:19.895951 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:19.895917 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" event={"ID":"64e89b6e-0836-4a95-adfb-fb359827d28e","Type":"ContainerStarted","Data":"5127c968974e268c5cd26f8963b1b2e79d8e3ff152bd4f8dbb4505ad2b67a2f3"} Oct 31 01:03:19.896344 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:19.895976 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:20.014187 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:20.014124 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" podStartSLOduration=2.031338968 podStartE2EDuration="3.014103972s" totalImagesPullingTime="982.765004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:20.012745261 +0000 UTC m=+3632.488103684" watchObservedRunningTime="2026-10-31 01:03:20.014103972 +0000 UTC m=+3632.489462397" Oct 31 01:03:25.907983 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:25.907951 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:03:32.729222 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:03:32.729186 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:03:44.749705 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.749615 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6"] Oct 31 01:03:44.754955 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.754929 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:03:44.763957 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.763841 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6"] Oct 31 01:03:44.795810 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.795780 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee68c13-b7f1-4612-a57b-3b928d9d0019-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6\" (UID: \"bee68c13-b7f1-4612-a57b-3b928d9d0019\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:03:44.796134 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.796108 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwkj\" (UniqueName: \"kubernetes.io/projected/bee68c13-b7f1-4612-a57b-3b928d9d0019-kube-api-access-2bwkj\") pod \"payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6\" (UID: \"bee68c13-b7f1-4612-a57b-3b928d9d0019\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:03:44.807863 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.807837 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-395b06\"" Oct 31 01:03:44.807863 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.807850 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-395b06-dockercfg-lxlpc\"" Oct 31 01:03:44.897823 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.897789 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwkj\" (UniqueName: \"kubernetes.io/projected/bee68c13-b7f1-4612-a57b-3b928d9d0019-kube-api-access-2bwkj\") pod \"payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6\" (UID: \"bee68c13-b7f1-4612-a57b-3b928d9d0019\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:03:44.898000 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.897884 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee68c13-b7f1-4612-a57b-3b928d9d0019-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6\" (UID: \"bee68c13-b7f1-4612-a57b-3b928d9d0019\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:03:44.898651 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.898585 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee68c13-b7f1-4612-a57b-3b928d9d0019-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6\" (UID: \"bee68c13-b7f1-4612-a57b-3b928d9d0019\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:03:44.959102 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:44.959064 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwkj\" (UniqueName: \"kubernetes.io/projected/bee68c13-b7f1-4612-a57b-3b928d9d0019-kube-api-access-2bwkj\") pod \"payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6\" (UID: \"bee68c13-b7f1-4612-a57b-3b928d9d0019\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:03:45.071842 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:45.071751 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:03:45.459229 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:03:45.459199 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee68c13_b7f1_4612_a57b_3b928d9d0019.slice/crio-8d154efa2a663298c24d178f66efa5253a84e0c87cfdfdc582424deacdb5a5aa WatchSource:0}: Error finding container 8d154efa2a663298c24d178f66efa5253a84e0c87cfdfdc582424deacdb5a5aa: Status 404 returned error can't find the container with id 8d154efa2a663298c24d178f66efa5253a84e0c87cfdfdc582424deacdb5a5aa Oct 31 01:03:45.459370 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:45.459349 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6"] Oct 31 01:03:46.110723 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:46.110684 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" event={"ID":"bee68c13-b7f1-4612-a57b-3b928d9d0019","Type":"ContainerStarted","Data":"489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018"} Oct 31 01:03:46.111107 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:46.110762 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" event={"ID":"bee68c13-b7f1-4612-a57b-3b928d9d0019","Type":"ContainerStarted","Data":"8d154efa2a663298c24d178f66efa5253a84e0c87cfdfdc582424deacdb5a5aa"} Oct 31 01:03:46.111107 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:46.110835 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:03:46.229812 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:46.229755 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" podStartSLOduration=2.22974154 podStartE2EDuration="2.22974154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:46.226942808 +0000 UTC m=+3658.702301230" watchObservedRunningTime="2026-10-31 01:03:46.22974154 +0000 UTC m=+3658.705099968" Oct 31 01:03:57.121215 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:03:57.121184 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:04:02.728666 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:04:02.728627 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:04:12.137763 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.137727 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9"] Oct 31 01:04:12.142153 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.142137 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.196280 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.196254 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-5498e8-serving-cert\"" Oct 31 01:04:12.239724 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.239692 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66md\" (UniqueName: \"kubernetes.io/projected/054abb30-24f5-4968-a793-a285ad5a7d43-kube-api-access-f66md\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.239933 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.239763 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-metrics-certs\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.239933 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.239891 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.240058 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.239947 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054abb30-24f5-4968-a793-a285ad5a7d43-openshift-service-ca\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.249631 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.249586 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9"] Oct 31 01:04:12.341625 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.341549 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66md\" (UniqueName: \"kubernetes.io/projected/054abb30-24f5-4968-a793-a285ad5a7d43-kube-api-access-f66md\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.341827 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.341688 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-metrics-certs\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.341827 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.341795 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.341961 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.341860 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054abb30-24f5-4968-a793-a285ad5a7d43-openshift-service-ca\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.341961 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:04:12.341935 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-serving-cert: secret "maas-api-e2e-explicit-5498e8-serving-cert" not found Oct 31 01:04:12.342048 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:04:12.342020 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls podName:054abb30-24f5-4968-a793-a285ad5a7d43 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:12.841998518 +0000 UTC m=+3685.317356947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls") pod "maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" (UID: "054abb30-24f5-4968-a793-a285ad5a7d43") : secret "maas-api-e2e-explicit-5498e8-serving-cert" not found Oct 31 01:04:12.342581 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.342552 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054abb30-24f5-4968-a793-a285ad5a7d43-openshift-service-ca\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.344294 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.344268 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-metrics-certs\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.404049 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.403944 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66md\" (UniqueName: \"kubernetes.io/projected/054abb30-24f5-4968-a793-a285ad5a7d43-kube-api-access-f66md\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.848551 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.848452 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:12.851170 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:12.851140 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls\") pod \"maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:13.056647 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:13.056583 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:13.438400 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:13.438367 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9"] Oct 31 01:04:13.443182 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:04:13.443145 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054abb30_24f5_4968_a793_a285ad5a7d43.slice/crio-2fd7a4f6cc32632fcb85aa014e75eabeac36d5105e80196c0961d18fecc93474 WatchSource:0}: Error finding container 2fd7a4f6cc32632fcb85aa014e75eabeac36d5105e80196c0961d18fecc93474: Status 404 returned error can't find the container with id 2fd7a4f6cc32632fcb85aa014e75eabeac36d5105e80196c0961d18fecc93474 Oct 31 01:04:13.444816 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:13.444791 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:04:14.328571 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:14.328534 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" event={"ID":"054abb30-24f5-4968-a793-a285ad5a7d43","Type":"ContainerStarted","Data":"2fd7a4f6cc32632fcb85aa014e75eabeac36d5105e80196c0961d18fecc93474"} Oct 31 01:04:15.337854 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:15.337817 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" event={"ID":"054abb30-24f5-4968-a793-a285ad5a7d43","Type":"ContainerStarted","Data":"8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f"} Oct 31 01:04:15.338282 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:15.337883 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:15.457280 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:15.457222 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" podStartSLOduration=2.511563338 podStartE2EDuration="3.457206867s" totalImagesPullingTime="945.643529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:15.454838617 +0000 UTC m=+3687.930197039" watchObservedRunningTime="2026-10-31 01:04:15.457206867 +0000 UTC m=+3687.932565290" Oct 31 01:04:21.351846 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:04:21.351812 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:04:32.728680 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:04:32.728641 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:05:02.733528 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:05:02.733438 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:05:12.612939 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:12.612906 4969 patch_prober.go:29] interesting pod/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 01:05:12.613365 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:12.612965 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 01:05:12.793379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:12.793344 4969 generic.go:356] "Generic (PLEG): container finished" podID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerID="15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472" exitCode=0 Oct 31 01:05:12.793546 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:12.793388 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" event={"ID":"cf2ca7f1-d050-4b15-917b-e901168b2199","Type":"ContainerDied","Data":"15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472"} Oct 31 01:05:12.793817 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:12.793798 4969 scope.go:122] "RemoveContainer" containerID="15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472" Oct 31 01:05:12.795390 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:12.795362 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:05:12.795461 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:12.795447 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:05:12.795505 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:12.795482 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:05:13.814035 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:13.813998 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" event={"ID":"cf2ca7f1-d050-4b15-917b-e901168b2199","Type":"ContainerStarted","Data":"0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599"} Oct 31 01:05:14.000618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:13.987970 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:05:14.000618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:13.988091 4969 patch_prober.go:29] interesting pod/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 01:05:14.000618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:13.988147 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 01:05:14.997618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:14.988902 4969 patch_prober.go:29] interesting pod/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 01:05:14.997618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:14.988973 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 01:05:16.001751 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:16.001706 4969 patch_prober.go:29] interesting pod/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 01:05:16.002213 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:16.001772 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 01:05:16.998616 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:16.997971 4969 patch_prober.go:29] interesting pod/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 01:05:16.998616 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:16.998029 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 01:05:17.991089 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:17.991042 4969 patch_prober.go:29] interesting pod/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 01:05:17.991785 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:17.991745 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 01:05:18.984620 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:18.981932 4969 patch_prober.go:29] interesting pod/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 31 01:05:18.984620 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:18.982004 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 31 01:05:19.987587 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:19.987558 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:05:19.988034 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:19.988005 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:05:19.988802 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:19.988769 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:05:24.225947 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.225908 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx"] Oct 31 01:05:24.230186 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.230166 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.284324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.284290 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-395b06-serving-cert\"" Oct 31 01:05:24.289149 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.289119 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx"] Oct 31 01:05:24.324003 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.323970 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vhz\" (UniqueName: \"kubernetes.io/projected/e7d0357c-9c5f-40c1-87ba-9190a0de2551-kube-api-access-z6vhz\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.324145 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.324089 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-metrics-certs\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.324212 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.324150 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d0357c-9c5f-40c1-87ba-9190a0de2551-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.324275 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.324262 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.425365 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.425333 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.425545 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.425474 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vhz\" (UniqueName: \"kubernetes.io/projected/e7d0357c-9c5f-40c1-87ba-9190a0de2551-kube-api-access-z6vhz\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.425545 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:05:24.425491 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-serving-cert: secret "maas-api-e2e-extmod-b-395b06-serving-cert" not found Oct 31 01:05:24.425705 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.425545 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-metrics-certs\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.425705 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:05:24.425564 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls podName:e7d0357c-9c5f-40c1-87ba-9190a0de2551 nodeName:}" failed. No retries permitted until 2026-10-31 01:05:24.925543723 +0000 UTC m=+3757.400902140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls") pod "maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" (UID: "e7d0357c-9c5f-40c1-87ba-9190a0de2551") : secret "maas-api-e2e-extmod-b-395b06-serving-cert" not found Oct 31 01:05:24.425705 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.425646 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d0357c-9c5f-40c1-87ba-9190a0de2551-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.426334 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.426305 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d0357c-9c5f-40c1-87ba-9190a0de2551-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.428342 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.428321 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-metrics-certs\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.487255 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.487191 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vhz\" (UniqueName: \"kubernetes.io/projected/e7d0357c-9c5f-40c1-87ba-9190a0de2551-kube-api-access-z6vhz\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.933865 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.933837 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:24.936421 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:24.936397 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls\") pod \"maas-api-e2e-extmod-b-395b06-856679976c-rkpnx\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:25.144332 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:25.144302 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:25.527220 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:05:25.527178 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d0357c_9c5f_40c1_87ba_9190a0de2551.slice/crio-6a6ed4bb3451d782f88b0f216104567c9a3b985c88766b98a04087104078af5b WatchSource:0}: Error finding container 6a6ed4bb3451d782f88b0f216104567c9a3b985c88766b98a04087104078af5b: Status 404 returned error can't find the container with id 6a6ed4bb3451d782f88b0f216104567c9a3b985c88766b98a04087104078af5b Oct 31 01:05:25.527679 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:25.527569 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx"] Oct 31 01:05:25.909053 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:25.909015 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" event={"ID":"e7d0357c-9c5f-40c1-87ba-9190a0de2551","Type":"ContainerStarted","Data":"6a6ed4bb3451d782f88b0f216104567c9a3b985c88766b98a04087104078af5b"} Oct 31 01:05:26.919073 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:26.919033 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" event={"ID":"e7d0357c-9c5f-40c1-87ba-9190a0de2551","Type":"ContainerStarted","Data":"25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658"} Oct 31 01:05:26.919523 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:26.919112 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:27.042694 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:27.042644 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" podStartSLOduration=2.196742986 podStartE2EDuration="3.042629855s" totalImagesPullingTime="845.886869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:27.037133441 +0000 UTC m=+3759.512491863" watchObservedRunningTime="2026-10-31 01:05:27.042629855 +0000 UTC m=+3759.517988281" Oct 31 01:05:32.729738 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:05:32.729699 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:05:32.933883 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:32.933858 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:05:53.071531 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:53.071493 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d"] Oct 31 01:05:58.071929 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.071846 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" podUID="62a5be49-af4b-45cf-9f8d-8fc49eb4ba75" containerName="payload-pre-processing" containerID="cri-o://3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8" gracePeriod=25 Oct 31 01:05:58.192541 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.192510 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9"] Oct 31 01:05:58.192758 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.192730 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" podUID="054abb30-24f5-4968-a793-a285ad5a7d43" containerName="maas-api" containerID="cri-o://8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f" gracePeriod=30 Oct 31 01:05:58.376292 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.376262 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:05:58.484850 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.484816 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-plugins-config-volume\") pod \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\" (UID: \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\") " Oct 31 01:05:58.484979 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.484963 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-kube-api-access-vcww8\" (UniqueName: \"kubernetes.io/projected/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-kube-api-access-vcww8\") pod \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\" (UID: \"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75\") " Oct 31 01:05:58.485253 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.485232 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-plugins-config-volume" pod "62a5be49-af4b-45cf-9f8d-8fc49eb4ba75" (UID: "62a5be49-af4b-45cf-9f8d-8fc49eb4ba75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:58.485516 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.485496 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:58.487171 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.487149 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-kube-api-access-vcww8" pod "62a5be49-af4b-45cf-9f8d-8fc49eb4ba75" (UID: "62a5be49-af4b-45cf-9f8d-8fc49eb4ba75"). InnerVolumeSpecName "kube-api-access-vcww8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:58.491119 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.491101 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:05:58.590654 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.587048 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054abb30-24f5-4968-a793-a285ad5a7d43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054abb30-24f5-4968-a793-a285ad5a7d43-openshift-service-ca\") pod \"054abb30-24f5-4968-a793-a285ad5a7d43\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " Oct 31 01:05:58.590654 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.587124 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-metrics-certs\") pod \"054abb30-24f5-4968-a793-a285ad5a7d43\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " Oct 31 01:05:58.590654 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.587266 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054abb30-24f5-4968-a793-a285ad5a7d43-kube-api-access-f66md\" (UniqueName: \"kubernetes.io/projected/054abb30-24f5-4968-a793-a285ad5a7d43-kube-api-access-f66md\") pod \"054abb30-24f5-4968-a793-a285ad5a7d43\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " Oct 31 01:05:58.590654 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.587378 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls\") pod \"054abb30-24f5-4968-a793-a285ad5a7d43\" (UID: \"054abb30-24f5-4968-a793-a285ad5a7d43\") " Oct 31 01:05:58.590654 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.587883 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcww8\" (UniqueName: \"kubernetes.io/projected/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75-kube-api-access-vcww8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:58.591918 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.591888 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054abb30-24f5-4968-a793-a285ad5a7d43-openshift-service-ca" pod "054abb30-24f5-4968-a793-a285ad5a7d43" (UID: "054abb30-24f5-4968-a793-a285ad5a7d43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:58.593187 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.593158 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-metrics-certs" pod "054abb30-24f5-4968-a793-a285ad5a7d43" (UID: "054abb30-24f5-4968-a793-a285ad5a7d43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:58.594924 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.594856 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls" pod "054abb30-24f5-4968-a793-a285ad5a7d43" (UID: "054abb30-24f5-4968-a793-a285ad5a7d43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:58.607673 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.601337 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054abb30-24f5-4968-a793-a285ad5a7d43-kube-api-access-f66md" pod "054abb30-24f5-4968-a793-a285ad5a7d43" (UID: "054abb30-24f5-4968-a793-a285ad5a7d43"). InnerVolumeSpecName "kube-api-access-f66md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:58.689151 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.689118 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66md\" (UniqueName: \"kubernetes.io/projected/054abb30-24f5-4968-a793-a285ad5a7d43-kube-api-access-f66md\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:58.689151 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.689146 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:58.689151 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.689156 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054abb30-24f5-4968-a793-a285ad5a7d43-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:58.689384 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:58.689164 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054abb30-24f5-4968-a793-a285ad5a7d43-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:05:59.195554 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.195513 4969 generic.go:356] "Generic (PLEG): container finished" podID="054abb30-24f5-4968-a793-a285ad5a7d43" containerID="8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f" exitCode=0 Oct 31 01:05:59.195977 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.195565 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" Oct 31 01:05:59.195977 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.195587 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" event={"ID":"054abb30-24f5-4968-a793-a285ad5a7d43","Type":"ContainerDied","Data":"8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f"} Oct 31 01:05:59.195977 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.195665 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9" event={"ID":"054abb30-24f5-4968-a793-a285ad5a7d43","Type":"ContainerDied","Data":"2fd7a4f6cc32632fcb85aa014e75eabeac36d5105e80196c0961d18fecc93474"} Oct 31 01:05:59.195977 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.195689 4969 scope.go:122] "RemoveContainer" containerID="8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f" Oct 31 01:05:59.197372 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.197348 4969 generic.go:356] "Generic (PLEG): container finished" podID="62a5be49-af4b-45cf-9f8d-8fc49eb4ba75" containerID="3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8" exitCode=0 Oct 31 01:05:59.197449 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.197391 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" Oct 31 01:05:59.197504 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.197442 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" event={"ID":"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75","Type":"ContainerDied","Data":"3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8"} Oct 31 01:05:59.197504 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.197481 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d" event={"ID":"62a5be49-af4b-45cf-9f8d-8fc49eb4ba75","Type":"ContainerDied","Data":"5fe3c26734aa74d4ce6e55ff9588ee12f88b14b6070625a109e32ad6e72e1606"} Oct 31 01:05:59.209224 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.209203 4969 scope.go:122] "RemoveContainer" containerID="8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f" Oct 31 01:05:59.209492 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:05:59.209473 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f\": container with ID starting with 8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f not found: ID does not exist" containerID="8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f" Oct 31 01:05:59.209543 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.209500 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f"} err="failed to get container status \"8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f\": rpc error: code = NotFound desc = could not find container \"8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f\": container with ID starting with 8fa1be66b117c8fcfe6cb6f913c60944ddfa38e96ed9d35652d72ea6aaee593f not found: ID does not exist" Oct 31 01:05:59.209543 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.209516 4969 scope.go:122] "RemoveContainer" containerID="3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8" Oct 31 01:05:59.220333 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.220315 4969 scope.go:122] "RemoveContainer" containerID="3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8" Oct 31 01:05:59.220638 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:05:59.220619 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8\": container with ID starting with 3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8 not found: ID does not exist" containerID="3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8" Oct 31 01:05:59.220713 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.220645 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8"} err="failed to get container status \"3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8\": rpc error: code = NotFound desc = could not find container \"3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8\": container with ID starting with 3660654a06677a512cb90c6d3f984ec2a150ee95d7ea528b2008fc84628ddfa8 not found: ID does not exist" Oct 31 01:05:59.364190 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.364077 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9"] Oct 31 01:05:59.413921 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.413893 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5498e8-5b5bf5785-24kz9"] Oct 31 01:05:59.524017 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.523987 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d"] Oct 31 01:05:59.531914 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.531887 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5498e8-bf6796d9f-hsl2d"] Oct 31 01:05:59.735586 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.735554 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054abb30-24f5-4968-a793-a285ad5a7d43" path="/var/lib/kubelet/pods/054abb30-24f5-4968-a793-a285ad5a7d43/volumes" Oct 31 01:05:59.736015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:05:59.735999 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a5be49-af4b-45cf-9f8d-8fc49eb4ba75" path="/var/lib/kubelet/pods/62a5be49-af4b-45cf-9f8d-8fc49eb4ba75/volumes" Oct 31 01:06:02.729806 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:06:02.729770 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:06:13.954216 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:13.954182 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg"] Oct 31 01:06:13.954714 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:13.954384 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" containerID="cri-o://0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599" gracePeriod=30 Oct 31 01:06:19.276715 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.276691 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:06:19.362814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.362782 4969 generic.go:356] "Generic (PLEG): container finished" podID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerID="0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599" exitCode=0 Oct 31 01:06:19.362991 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.362839 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" Oct 31 01:06:19.362991 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.362861 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" event={"ID":"cf2ca7f1-d050-4b15-917b-e901168b2199","Type":"ContainerDied","Data":"0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599"} Oct 31 01:06:19.362991 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.362914 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg" event={"ID":"cf2ca7f1-d050-4b15-917b-e901168b2199","Type":"ContainerDied","Data":"68c22127e092b0267fe68923d10187afbbe986bc1b3f2299734badf5c760ced6"} Oct 31 01:06:19.362991 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.362938 4969 scope.go:122] "RemoveContainer" containerID="0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599" Oct 31 01:06:19.363388 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.363358 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-podinfo\") pod \"cf2ca7f1-d050-4b15-917b-e901168b2199\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " Oct 31 01:06:19.363529 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.363418 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-data\") pod \"cf2ca7f1-d050-4b15-917b-e901168b2199\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " Oct 31 01:06:19.363529 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.363470 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-envoy\") pod \"cf2ca7f1-d050-4b15-917b-e901168b2199\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " Oct 31 01:06:19.363675 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.363658 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-token\") pod \"cf2ca7f1-d050-4b15-917b-e901168b2199\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " Oct 31 01:06:19.363753 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.363735 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-kube-api-access-mndf8\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-kube-api-access-mndf8\") pod \"cf2ca7f1-d050-4b15-917b-e901168b2199\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " Oct 31 01:06:19.363856 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.363805 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-data" pod "cf2ca7f1-d050-4b15-917b-e901168b2199" (UID: "cf2ca7f1-d050-4b15-917b-e901168b2199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:19.363931 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.363862 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2ca7f1-d050-4b15-917b-e901168b2199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ca7f1-d050-4b15-917b-e901168b2199-istiod-ca-cert\") pod \"cf2ca7f1-d050-4b15-917b-e901168b2199\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " Oct 31 01:06:19.363931 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.363925 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-certs\") pod \"cf2ca7f1-d050-4b15-917b-e901168b2199\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " Oct 31 01:06:19.364051 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.363969 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-credential-socket\") pod \"cf2ca7f1-d050-4b15-917b-e901168b2199\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " Oct 31 01:06:19.364051 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.364021 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-socket\") pod \"cf2ca7f1-d050-4b15-917b-e901168b2199\" (UID: \"cf2ca7f1-d050-4b15-917b-e901168b2199\") " Oct 31 01:06:19.364288 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.364240 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-certs" pod "cf2ca7f1-d050-4b15-917b-e901168b2199" (UID: "cf2ca7f1-d050-4b15-917b-e901168b2199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:19.364517 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.364349 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2ca7f1-d050-4b15-917b-e901168b2199-istiod-ca-cert" pod "cf2ca7f1-d050-4b15-917b-e901168b2199" (UID: "cf2ca7f1-d050-4b15-917b-e901168b2199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:19.364720 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.364700 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.364815 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.364743 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.364815 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.364700 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-credential-socket" pod "cf2ca7f1-d050-4b15-917b-e901168b2199" (UID: "cf2ca7f1-d050-4b15-917b-e901168b2199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:19.364815 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.364759 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-socket" pod "cf2ca7f1-d050-4b15-917b-e901168b2199" (UID: "cf2ca7f1-d050-4b15-917b-e901168b2199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:19.366285 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.366255 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-kube-api-access-mndf8" pod "cf2ca7f1-d050-4b15-917b-e901168b2199" (UID: "cf2ca7f1-d050-4b15-917b-e901168b2199"). InnerVolumeSpecName "kube-api-access-mndf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:19.366626 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.366565 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-envoy" pod "cf2ca7f1-d050-4b15-917b-e901168b2199" (UID: "cf2ca7f1-d050-4b15-917b-e901168b2199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:06:19.366741 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.366674 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-token" pod "cf2ca7f1-d050-4b15-917b-e901168b2199" (UID: "cf2ca7f1-d050-4b15-917b-e901168b2199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:19.367291 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.367269 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-podinfo" pod "cf2ca7f1-d050-4b15-917b-e901168b2199" (UID: "cf2ca7f1-d050-4b15-917b-e901168b2199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:06:19.396073 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.396053 4969 scope.go:122] "RemoveContainer" containerID="15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472" Oct 31 01:06:19.407769 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.407749 4969 scope.go:122] "RemoveContainer" containerID="0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599" Oct 31 01:06:19.408063 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:06:19.408044 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599\": container with ID starting with 0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599 not found: ID does not exist" containerID="0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599" Oct 31 01:06:19.408111 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.408075 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599"} err="failed to get container status \"0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599\": rpc error: code = NotFound desc = could not find container \"0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599\": container with ID starting with 0dee950137388dd746f994bcfa6dae8ff219646a2efbc71578de6148eee56599 not found: ID does not exist" Oct 31 01:06:19.408111 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.408096 4969 scope.go:122] "RemoveContainer" containerID="15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472" Oct 31 01:06:19.408344 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:06:19.408325 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472\": container with ID starting with 15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472 not found: ID does not exist" containerID="15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472" Oct 31 01:06:19.408387 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.408352 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472"} err="failed to get container status \"15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472\": rpc error: code = NotFound desc = could not find container \"15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472\": container with ID starting with 15e0dd07d73c4ff405113fce7ec04f23c8a9a657b9144dc6b487af92f8ad5472 not found: ID does not exist" Oct 31 01:06:19.465812 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.465779 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.465812 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.465805 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.465812 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.465816 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.466042 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.465824 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.466042 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.465833 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.466042 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.465841 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mndf8\" (UniqueName: \"kubernetes.io/projected/cf2ca7f1-d050-4b15-917b-e901168b2199-kube-api-access-mndf8\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.466042 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.465851 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ca7f1-d050-4b15-917b-e901168b2199-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:19.808751 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.808710 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg"] Oct 31 01:06:19.813706 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:19.813674 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2aebbd-openshift-default-68cdbb8f-jfmlg"] Oct 31 01:06:21.736184 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:21.736149 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" path="/var/lib/kubelet/pods/cf2ca7f1-d050-4b15-917b-e901168b2199/volumes" Oct 31 01:06:32.728165 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:06:32.728128 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:06:36.369543 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.369508 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24"] Oct 31 01:06:36.370033 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.369742 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" podUID="64e89b6e-0836-4a95-adfb-fb359827d28e" containerName="maas-api" containerID="cri-o://5127c968974e268c5cd26f8963b1b2e79d8e3ff152bd4f8dbb4505ad2b67a2f3" gracePeriod=30 Oct 31 01:06:36.540055 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.540023 4969 generic.go:356] "Generic (PLEG): container finished" podID="64e89b6e-0836-4a95-adfb-fb359827d28e" containerID="5127c968974e268c5cd26f8963b1b2e79d8e3ff152bd4f8dbb4505ad2b67a2f3" exitCode=0 Oct 31 01:06:36.540218 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.540091 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" event={"ID":"64e89b6e-0836-4a95-adfb-fb359827d28e","Type":"ContainerDied","Data":"5127c968974e268c5cd26f8963b1b2e79d8e3ff152bd4f8dbb4505ad2b67a2f3"} Oct 31 01:06:36.655998 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.655972 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:06:36.792761 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.790701 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e89b6e-0836-4a95-adfb-fb359827d28e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89b6e-0836-4a95-adfb-fb359827d28e-openshift-service-ca\") pod \"64e89b6e-0836-4a95-adfb-fb359827d28e\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " Oct 31 01:06:36.792761 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.790820 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls\") pod \"64e89b6e-0836-4a95-adfb-fb359827d28e\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " Oct 31 01:06:36.792761 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.790974 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-metrics-certs\") pod \"64e89b6e-0836-4a95-adfb-fb359827d28e\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " Oct 31 01:06:36.792761 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.791115 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e89b6e-0836-4a95-adfb-fb359827d28e-kube-api-access-6dnvb\" (UniqueName: \"kubernetes.io/projected/64e89b6e-0836-4a95-adfb-fb359827d28e-kube-api-access-6dnvb\") pod \"64e89b6e-0836-4a95-adfb-fb359827d28e\" (UID: \"64e89b6e-0836-4a95-adfb-fb359827d28e\") " Oct 31 01:06:36.792761 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.791109 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e89b6e-0836-4a95-adfb-fb359827d28e-openshift-service-ca" pod "64e89b6e-0836-4a95-adfb-fb359827d28e" (UID: "64e89b6e-0836-4a95-adfb-fb359827d28e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:36.792761 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.791651 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89b6e-0836-4a95-adfb-fb359827d28e-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:36.797926 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.796554 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-metrics-certs" pod "64e89b6e-0836-4a95-adfb-fb359827d28e" (UID: "64e89b6e-0836-4a95-adfb-fb359827d28e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:36.797926 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.796926 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e89b6e-0836-4a95-adfb-fb359827d28e-kube-api-access-6dnvb" pod "64e89b6e-0836-4a95-adfb-fb359827d28e" (UID: "64e89b6e-0836-4a95-adfb-fb359827d28e"). InnerVolumeSpecName "kube-api-access-6dnvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:36.802749 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.802718 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls" pod "64e89b6e-0836-4a95-adfb-fb359827d28e" (UID: "64e89b6e-0836-4a95-adfb-fb359827d28e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:36.892721 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.892623 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:36.892721 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.892674 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dnvb\" (UniqueName: \"kubernetes.io/projected/64e89b6e-0836-4a95-adfb-fb359827d28e-kube-api-access-6dnvb\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:36.892721 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:36.892690 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64e89b6e-0836-4a95-adfb-fb359827d28e-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:37.549851 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:37.549810 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" Oct 31 01:06:37.550289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:37.549809 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24" event={"ID":"64e89b6e-0836-4a95-adfb-fb359827d28e","Type":"ContainerDied","Data":"9d4fe0ea6434e22e2c3a1645deb6aae1ad597290644f862f8f81a75c9118448f"} Oct 31 01:06:37.550289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:37.549980 4969 scope.go:122] "RemoveContainer" containerID="5127c968974e268c5cd26f8963b1b2e79d8e3ff152bd4f8dbb4505ad2b67a2f3" Oct 31 01:06:37.737405 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:37.737373 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24"] Oct 31 01:06:37.779047 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:37.779009 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cc6150-758c8dfc8c-xmt24"] Oct 31 01:06:39.736182 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:06:39.736151 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e89b6e-0836-4a95-adfb-fb359827d28e" path="/var/lib/kubelet/pods/64e89b6e-0836-4a95-adfb-fb359827d28e/volumes" Oct 31 01:07:01.715380 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:01.715348 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk"] Oct 31 01:07:01.822834 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:01.822800 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42"] Oct 31 01:07:02.728584 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:02.728547 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:07:06.715989 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:06.715950 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" podUID="a9a14c40-5fed-42f8-b093-f167eefd7bbd" containerName="payload-processing" containerID="cri-o://e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a" gracePeriod=25 Oct 31 01:07:06.823740 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:06.823704 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" podUID="10afad44-9b46-438a-8de0-affbdb0e8405" containerName="payload-pre-processing" containerID="cri-o://4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29" gracePeriod=25 Oct 31 01:07:06.993880 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:06.993853 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 01:07:07.083503 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.083476 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a14c40-5fed-42f8-b093-f167eefd7bbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a14c40-5fed-42f8-b093-f167eefd7bbd-plugins-config-volume\") pod \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\" (UID: \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\") " Oct 31 01:07:07.083647 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.083581 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a14c40-5fed-42f8-b093-f167eefd7bbd-kube-api-access-hcdhv\" (UniqueName: \"kubernetes.io/projected/a9a14c40-5fed-42f8-b093-f167eefd7bbd-kube-api-access-hcdhv\") pod \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\" (UID: \"a9a14c40-5fed-42f8-b093-f167eefd7bbd\") " Oct 31 01:07:07.083917 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.083893 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a14c40-5fed-42f8-b093-f167eefd7bbd-plugins-config-volume" pod "a9a14c40-5fed-42f8-b093-f167eefd7bbd" (UID: "a9a14c40-5fed-42f8-b093-f167eefd7bbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:07.084071 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.084055 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a14c40-5fed-42f8-b093-f167eefd7bbd-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.085895 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.085875 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a14c40-5fed-42f8-b093-f167eefd7bbd-kube-api-access-hcdhv" pod "a9a14c40-5fed-42f8-b093-f167eefd7bbd" (UID: "a9a14c40-5fed-42f8-b093-f167eefd7bbd"). InnerVolumeSpecName "kube-api-access-hcdhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:07.100777 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.100756 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 01:07:07.184920 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.184887 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10afad44-9b46-438a-8de0-affbdb0e8405-kube-api-access-6gsgf\" (UniqueName: \"kubernetes.io/projected/10afad44-9b46-438a-8de0-affbdb0e8405-kube-api-access-6gsgf\") pod \"10afad44-9b46-438a-8de0-affbdb0e8405\" (UID: \"10afad44-9b46-438a-8de0-affbdb0e8405\") " Oct 31 01:07:07.185099 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.184956 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10afad44-9b46-438a-8de0-affbdb0e8405-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10afad44-9b46-438a-8de0-affbdb0e8405-plugins-config-volume\") pod \"10afad44-9b46-438a-8de0-affbdb0e8405\" (UID: \"10afad44-9b46-438a-8de0-affbdb0e8405\") " Oct 31 01:07:07.185356 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.185332 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10afad44-9b46-438a-8de0-affbdb0e8405-plugins-config-volume" pod "10afad44-9b46-438a-8de0-affbdb0e8405" (UID: "10afad44-9b46-438a-8de0-affbdb0e8405"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:07.185398 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.185389 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcdhv\" (UniqueName: \"kubernetes.io/projected/a9a14c40-5fed-42f8-b093-f167eefd7bbd-kube-api-access-hcdhv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.187249 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.187231 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10afad44-9b46-438a-8de0-affbdb0e8405-kube-api-access-6gsgf" pod "10afad44-9b46-438a-8de0-affbdb0e8405" (UID: "10afad44-9b46-438a-8de0-affbdb0e8405"). InnerVolumeSpecName "kube-api-access-6gsgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:07.287032 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.286968 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10afad44-9b46-438a-8de0-affbdb0e8405-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.287032 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.286995 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gsgf\" (UniqueName: \"kubernetes.io/projected/10afad44-9b46-438a-8de0-affbdb0e8405-kube-api-access-6gsgf\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:07.779392 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.779360 4969 generic.go:356] "Generic (PLEG): container finished" podID="a9a14c40-5fed-42f8-b093-f167eefd7bbd" containerID="e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a" exitCode=0 Oct 31 01:07:07.779832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.779406 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" Oct 31 01:07:07.779832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.779445 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" event={"ID":"a9a14c40-5fed-42f8-b093-f167eefd7bbd","Type":"ContainerDied","Data":"e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a"} Oct 31 01:07:07.779832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.779502 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk" event={"ID":"a9a14c40-5fed-42f8-b093-f167eefd7bbd","Type":"ContainerDied","Data":"17cf65ebe479c38b1dd56f6fc3c126b0075f0d7274d60f206eb0e0770ca7a01e"} Oct 31 01:07:07.779832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.779534 4969 scope.go:122] "RemoveContainer" containerID="e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a" Oct 31 01:07:07.781902 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.781263 4969 generic.go:356] "Generic (PLEG): container finished" podID="10afad44-9b46-438a-8de0-affbdb0e8405" containerID="4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29" exitCode=0 Oct 31 01:07:07.781902 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.781327 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" Oct 31 01:07:07.781902 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.781296 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" event={"ID":"10afad44-9b46-438a-8de0-affbdb0e8405","Type":"ContainerDied","Data":"4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29"} Oct 31 01:07:07.781902 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.781476 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42" event={"ID":"10afad44-9b46-438a-8de0-affbdb0e8405","Type":"ContainerDied","Data":"33ec1c8fd5159c8eaa814051636521e58998cf490e230545207b893799d2bad4"} Oct 31 01:07:07.790734 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.790715 4969 scope.go:122] "RemoveContainer" containerID="e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a" Oct 31 01:07:07.791016 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:07.790990 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a\": container with ID starting with e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a not found: ID does not exist" containerID="e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a" Oct 31 01:07:07.791068 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.791023 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a"} err="failed to get container status \"e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a\": rpc error: code = NotFound desc = could not find container \"e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a\": container with ID starting with e69bfb608fc2a6eb856816fa1a67e91d399c8e5a845d2ad75eb12c4aeeaa309a not found: ID does not exist" Oct 31 01:07:07.791068 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.791039 4969 scope.go:122] "RemoveContainer" containerID="4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29" Oct 31 01:07:07.803822 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.803801 4969 scope.go:122] "RemoveContainer" containerID="4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29" Oct 31 01:07:07.804084 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:07.804065 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29\": container with ID starting with 4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29 not found: ID does not exist" containerID="4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29" Oct 31 01:07:07.804131 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.804090 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29"} err="failed to get container status \"4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29\": rpc error: code = NotFound desc = could not find container \"4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29\": container with ID starting with 4b178fb08bf27b8fc65928f4e532324778617a15e0de5ed6772330cf768e4b29 not found: ID does not exist" Oct 31 01:07:07.945722 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.945689 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk"] Oct 31 01:07:07.995867 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:07.995797 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a7eb09-755bf9b8f5-gmnrk"] Oct 31 01:07:08.059331 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:08.059299 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42"] Oct 31 01:07:08.063343 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:08.063317 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a7eb09-ff54876c-kgl42"] Oct 31 01:07:09.734917 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:09.734884 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10afad44-9b46-438a-8de0-affbdb0e8405" path="/var/lib/kubelet/pods/10afad44-9b46-438a-8de0-affbdb0e8405/volumes" Oct 31 01:07:09.735304 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:09.735264 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a14c40-5fed-42f8-b093-f167eefd7bbd" path="/var/lib/kubelet/pods/a9a14c40-5fed-42f8-b093-f167eefd7bbd/volumes" Oct 31 01:07:23.005785 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:23.005753 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf"] Oct 31 01:07:23.006281 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:23.005992 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerName="istio-proxy" containerID="cri-o://08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0" gracePeriod=30 Oct 31 01:07:23.107443 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:23.107412 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm"] Oct 31 01:07:23.107705 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:23.107679 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" containerID="cri-o://bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce" gracePeriod=30 Oct 31 01:07:28.284392 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.284325 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 01:07:28.400417 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.400391 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 01:07:28.456972 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.456945 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-token\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-token\") pod \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " Oct 31 01:07:28.457152 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457085 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-envoy\") pod \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " Oct 31 01:07:28.457152 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457118 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-data\") pod \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " Oct 31 01:07:28.457286 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457189 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-podinfo\") pod \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " Oct 31 01:07:28.457286 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457221 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istiod-ca-cert\") pod \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " Oct 31 01:07:28.457286 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457260 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-certs\") pod \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " Oct 31 01:07:28.457462 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457300 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-socket\") pod \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " Oct 31 01:07:28.457462 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457372 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-credential-socket\") pod \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " Oct 31 01:07:28.457462 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457416 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-kube-api-access-w2ldj\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-kube-api-access-w2ldj\") pod \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\" (UID: \"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2\") " Oct 31 01:07:28.457462 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457432 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-data" pod "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" (UID: "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.457715 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457634 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-socket" pod "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" (UID: "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.457780 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.457673 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istiod-ca-cert" pod "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" (UID: "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:28.458115 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.458024 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.458115 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.458053 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.458115 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.458070 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.458521 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.458121 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-credential-socket" pod "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" (UID: "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.458521 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.458176 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-certs" pod "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" (UID: "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.460084 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.460036 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-envoy" pod "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" (UID: "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.460207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.460177 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-kube-api-access-w2ldj" pod "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" (UID: "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2"). InnerVolumeSpecName "kube-api-access-w2ldj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:28.460441 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.460417 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-token" pod "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" (UID: "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:28.460441 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.460415 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-podinfo" pod "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" (UID: "4c5a47e4-abdc-40ef-96c8-7777e2bca4b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:07:28.559329 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559253 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-certs\") pod \"7697441d-50ec-428c-90ef-1286c6498dc6\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " Oct 31 01:07:28.559329 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559315 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-kube-api-access-w54v6\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-kube-api-access-w54v6\") pod \"7697441d-50ec-428c-90ef-1286c6498dc6\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " Oct 31 01:07:28.559558 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559346 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-socket\") pod \"7697441d-50ec-428c-90ef-1286c6498dc6\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " Oct 31 01:07:28.559558 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559367 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-credential-socket\") pod \"7697441d-50ec-428c-90ef-1286c6498dc6\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " Oct 31 01:07:28.559558 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559469 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-envoy\") pod \"7697441d-50ec-428c-90ef-1286c6498dc6\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " Oct 31 01:07:28.559558 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559504 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-data\") pod \"7697441d-50ec-428c-90ef-1286c6498dc6\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " Oct 31 01:07:28.559558 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559531 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-certs" pod "7697441d-50ec-428c-90ef-1286c6498dc6" (UID: "7697441d-50ec-428c-90ef-1286c6498dc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.559875 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559572 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7697441d-50ec-428c-90ef-1286c6498dc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7697441d-50ec-428c-90ef-1286c6498dc6-istio-podinfo\") pod \"7697441d-50ec-428c-90ef-1286c6498dc6\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " Oct 31 01:07:28.559875 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559608 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-istio-token\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-istio-token\") pod \"7697441d-50ec-428c-90ef-1286c6498dc6\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " Oct 31 01:07:28.559875 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559651 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7697441d-50ec-428c-90ef-1286c6498dc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7697441d-50ec-428c-90ef-1286c6498dc6-istiod-ca-cert\") pod \"7697441d-50ec-428c-90ef-1286c6498dc6\" (UID: \"7697441d-50ec-428c-90ef-1286c6498dc6\") " Oct 31 01:07:28.559875 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559680 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-socket" pod "7697441d-50ec-428c-90ef-1286c6498dc6" (UID: "7697441d-50ec-428c-90ef-1286c6498dc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.559875 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559707 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-credential-socket" pod "7697441d-50ec-428c-90ef-1286c6498dc6" (UID: "7697441d-50ec-428c-90ef-1286c6498dc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.560162 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.559950 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-data" pod "7697441d-50ec-428c-90ef-1286c6498dc6" (UID: "7697441d-50ec-428c-90ef-1286c6498dc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.560230 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560178 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560230 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560200 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560230 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560215 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560233 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560247 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560261 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2ldj\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-kube-api-access-w2ldj\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560277 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560290 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560303 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560316 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.560564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.560425 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7697441d-50ec-428c-90ef-1286c6498dc6-istiod-ca-cert" pod "7697441d-50ec-428c-90ef-1286c6498dc6" (UID: "7697441d-50ec-428c-90ef-1286c6498dc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:28.561745 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.561723 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-kube-api-access-w54v6" pod "7697441d-50ec-428c-90ef-1286c6498dc6" (UID: "7697441d-50ec-428c-90ef-1286c6498dc6"). InnerVolumeSpecName "kube-api-access-w54v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:28.562038 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.562015 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-envoy" pod "7697441d-50ec-428c-90ef-1286c6498dc6" (UID: "7697441d-50ec-428c-90ef-1286c6498dc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:28.562126 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.562036 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7697441d-50ec-428c-90ef-1286c6498dc6-istio-podinfo" pod "7697441d-50ec-428c-90ef-1286c6498dc6" (UID: "7697441d-50ec-428c-90ef-1286c6498dc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:07:28.562126 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.562037 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-istio-token" pod "7697441d-50ec-428c-90ef-1286c6498dc6" (UID: "7697441d-50ec-428c-90ef-1286c6498dc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:28.661285 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.661253 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7697441d-50ec-428c-90ef-1286c6498dc6-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.661285 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.661284 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7697441d-50ec-428c-90ef-1286c6498dc6-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.661496 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.661298 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.661496 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.661310 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7697441d-50ec-428c-90ef-1286c6498dc6-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.661496 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.661323 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w54v6\" (UniqueName: \"kubernetes.io/projected/7697441d-50ec-428c-90ef-1286c6498dc6-kube-api-access-w54v6\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:28.961646 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.961578 4969 generic.go:356] "Generic (PLEG): container finished" podID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerID="08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0" exitCode=0 Oct 31 01:07:28.961646 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.961647 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" Oct 31 01:07:28.961940 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.961668 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" event={"ID":"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2","Type":"ContainerDied","Data":"08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0"} Oct 31 01:07:28.961940 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.961723 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf" event={"ID":"4c5a47e4-abdc-40ef-96c8-7777e2bca4b2","Type":"ContainerDied","Data":"45b23b63100c9ba965dffa32f4d20d0ac7631ad7f28fbff858077194dc3268b2"} Oct 31 01:07:28.961940 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.961746 4969 scope.go:122] "RemoveContainer" containerID="08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0" Oct 31 01:07:28.963807 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.963781 4969 generic.go:356] "Generic (PLEG): container finished" podID="7697441d-50ec-428c-90ef-1286c6498dc6" containerID="bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce" exitCode=0 Oct 31 01:07:28.963953 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.963819 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" Oct 31 01:07:28.963953 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.963856 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" event={"ID":"7697441d-50ec-428c-90ef-1286c6498dc6","Type":"ContainerDied","Data":"bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce"} Oct 31 01:07:28.963953 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.963899 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm" event={"ID":"7697441d-50ec-428c-90ef-1286c6498dc6","Type":"ContainerDied","Data":"9746d13f8ff3084e77fbfa26f8d76536cfc7002f51f14478e00cb7152be74d6c"} Oct 31 01:07:28.974154 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.974137 4969 scope.go:122] "RemoveContainer" containerID="08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0" Oct 31 01:07:28.974458 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:28.974434 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0\": container with ID starting with 08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0 not found: ID does not exist" containerID="08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0" Oct 31 01:07:28.974565 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.974464 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0"} err="failed to get container status \"08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0\": rpc error: code = NotFound desc = could not find container \"08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0\": container with ID starting with 08b37833d7a5cc6c89c46d688be9d381276348a77682e7aaa586fc8cea0e25a0 not found: ID does not exist" Oct 31 01:07:28.974565 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.974483 4969 scope.go:122] "RemoveContainer" containerID="bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce" Oct 31 01:07:28.984496 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.984468 4969 scope.go:122] "RemoveContainer" containerID="98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6" Oct 31 01:07:28.994069 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.994048 4969 scope.go:122] "RemoveContainer" containerID="bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce" Oct 31 01:07:28.994322 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:28.994306 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce\": container with ID starting with bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce not found: ID does not exist" containerID="bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce" Oct 31 01:07:28.994378 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.994329 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce"} err="failed to get container status \"bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce\": rpc error: code = NotFound desc = could not find container \"bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce\": container with ID starting with bf39cae82d53138d1dca3f7c4357c46bef5fd4bc70368fe964f064e68786abce not found: ID does not exist" Oct 31 01:07:28.994378 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.994347 4969 scope.go:122] "RemoveContainer" containerID="98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6" Oct 31 01:07:28.994605 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:28.994571 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6\": container with ID starting with 98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6 not found: ID does not exist" containerID="98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6" Oct 31 01:07:28.994671 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:28.994618 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6"} err="failed to get container status \"98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6\": rpc error: code = NotFound desc = could not find container \"98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6\": container with ID starting with 98f259ad48140db0dad2b32cbb79ec2d45df268365da098083216611df6853e6 not found: ID does not exist" Oct 31 01:07:29.127714 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:29.127682 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf"] Oct 31 01:07:29.177090 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:29.177063 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-mzjjf"] Oct 31 01:07:29.285957 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:29.285884 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm"] Oct 31 01:07:29.290600 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:29.290565 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-16b984-openshift-default-786bf945ff-m5bhm"] Oct 31 01:07:29.735299 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:29.735268 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" path="/var/lib/kubelet/pods/4c5a47e4-abdc-40ef-96c8-7777e2bca4b2/volumes" Oct 31 01:07:29.735826 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:29.735809 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" path="/var/lib/kubelet/pods/7697441d-50ec-428c-90ef-1286c6498dc6/volumes" Oct 31 01:07:32.728305 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:32.728269 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:07:35.420001 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:35.419969 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8"] Oct 31 01:07:40.198928 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.198896 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj"] Oct 31 01:07:40.199499 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.199170 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" podUID="c29a9d25-cb21-4358-a2a8-2644ffca19e3" containerName="maas-api" containerID="cri-o://9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7" gracePeriod=30 Oct 31 01:07:40.421061 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.420980 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" podUID="84e802b0-f574-4e5d-b253-c889fedc1716" containerName="payload-processing" containerID="cri-o://b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807" gracePeriod=25 Oct 31 01:07:40.472091 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.472064 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 01:07:40.583453 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.583213 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29a9d25-cb21-4358-a2a8-2644ffca19e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29a9d25-cb21-4358-a2a8-2644ffca19e3-openshift-service-ca\") pod \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " Oct 31 01:07:40.583453 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.583336 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-metrics-certs\") pod \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " Oct 31 01:07:40.583453 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.583414 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls\") pod \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " Oct 31 01:07:40.583936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.583521 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29a9d25-cb21-4358-a2a8-2644ffca19e3-kube-api-access-sgr2f\" (UniqueName: \"kubernetes.io/projected/c29a9d25-cb21-4358-a2a8-2644ffca19e3-kube-api-access-sgr2f\") pod \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\" (UID: \"c29a9d25-cb21-4358-a2a8-2644ffca19e3\") " Oct 31 01:07:40.585112 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.585074 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29a9d25-cb21-4358-a2a8-2644ffca19e3-openshift-service-ca" pod "c29a9d25-cb21-4358-a2a8-2644ffca19e3" (UID: "c29a9d25-cb21-4358-a2a8-2644ffca19e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:40.588295 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.588251 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29a9d25-cb21-4358-a2a8-2644ffca19e3-kube-api-access-sgr2f" pod "c29a9d25-cb21-4358-a2a8-2644ffca19e3" (UID: "c29a9d25-cb21-4358-a2a8-2644ffca19e3"). InnerVolumeSpecName "kube-api-access-sgr2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:40.591251 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.590259 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls" pod "c29a9d25-cb21-4358-a2a8-2644ffca19e3" (UID: "c29a9d25-cb21-4358-a2a8-2644ffca19e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:40.600986 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.600912 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-metrics-certs" pod "c29a9d25-cb21-4358-a2a8-2644ffca19e3" (UID: "c29a9d25-cb21-4358-a2a8-2644ffca19e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:40.685984 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.685944 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29a9d25-cb21-4358-a2a8-2644ffca19e3-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:40.685984 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.685976 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:40.685984 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.685989 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c29a9d25-cb21-4358-a2a8-2644ffca19e3-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:40.686227 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:40.686001 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgr2f\" (UniqueName: \"kubernetes.io/projected/c29a9d25-cb21-4358-a2a8-2644ffca19e3-kube-api-access-sgr2f\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:41.062650 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.062583 4969 generic.go:356] "Generic (PLEG): container finished" podID="c29a9d25-cb21-4358-a2a8-2644ffca19e3" containerID="9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7" exitCode=0 Oct 31 01:07:41.062869 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.062657 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" Oct 31 01:07:41.062869 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.062701 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" event={"ID":"c29a9d25-cb21-4358-a2a8-2644ffca19e3","Type":"ContainerDied","Data":"9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7"} Oct 31 01:07:41.062869 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.062758 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj" event={"ID":"c29a9d25-cb21-4358-a2a8-2644ffca19e3","Type":"ContainerDied","Data":"6d54ebd50b28c0f57148972709a7cf8aea49ebe2a75b3cd7ba3ed2a3cd9f6520"} Oct 31 01:07:41.062869 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.062786 4969 scope.go:122] "RemoveContainer" containerID="9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7" Oct 31 01:07:41.074656 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.074636 4969 scope.go:122] "RemoveContainer" containerID="9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7" Oct 31 01:07:41.074921 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:41.074897 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7\": container with ID starting with 9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7 not found: ID does not exist" containerID="9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7" Oct 31 01:07:41.074996 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.074932 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7"} err="failed to get container status \"9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7\": rpc error: code = NotFound desc = could not find container \"9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7\": container with ID starting with 9fee747c5ff03c46b84dc3f271e87daa3d2605661df4c7e0ae8baa2d288fe4b7 not found: ID does not exist" Oct 31 01:07:41.228660 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.228627 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj"] Oct 31 01:07:41.278200 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.278164 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a7eb09-655d64d7c8-sjgwj"] Oct 31 01:07:41.735580 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.735546 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29a9d25-cb21-4358-a2a8-2644ffca19e3" path="/var/lib/kubelet/pods/c29a9d25-cb21-4358-a2a8-2644ffca19e3/volumes" Oct 31 01:07:41.805721 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.805694 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q"] Oct 31 01:07:41.806272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806259 4969 cpu_manager.go:414] "containerMap: removing container" podUID="10afad44-9b46-438a-8de0-affbdb0e8405" containerName="payload-pre-processing" Oct 31 01:07:41.806327 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806274 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10afad44-9b46-438a-8de0-affbdb0e8405" containerName="payload-pre-processing" Oct 31 01:07:41.806327 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806285 4969 cpu_manager.go:414] "containerMap: removing container" podUID="64e89b6e-0836-4a95-adfb-fb359827d28e" containerName="maas-api" Oct 31 01:07:41.806327 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806291 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e89b6e-0836-4a95-adfb-fb359827d28e" containerName="maas-api" Oct 31 01:07:41.806327 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806307 4969 cpu_manager.go:414] "containerMap: removing container" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" Oct 31 01:07:41.806327 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806311 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" Oct 31 01:07:41.806327 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806319 4969 cpu_manager.go:414] "containerMap: removing container" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" Oct 31 01:07:41.806327 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806324 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806336 4969 cpu_manager.go:414] "containerMap: removing container" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806341 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806356 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a9a14c40-5fed-42f8-b093-f167eefd7bbd" containerName="payload-processing" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806361 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a14c40-5fed-42f8-b093-f167eefd7bbd" containerName="payload-processing" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806369 4969 cpu_manager.go:414] "containerMap: removing container" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806374 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806380 4969 cpu_manager.go:414] "containerMap: removing container" podUID="62a5be49-af4b-45cf-9f8d-8fc49eb4ba75" containerName="payload-pre-processing" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806385 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a5be49-af4b-45cf-9f8d-8fc49eb4ba75" containerName="payload-pre-processing" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806392 4969 cpu_manager.go:414] "containerMap: removing container" podUID="054abb30-24f5-4968-a793-a285ad5a7d43" containerName="maas-api" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806397 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054abb30-24f5-4968-a793-a285ad5a7d43" containerName="maas-api" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806406 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806410 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806432 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c29a9d25-cb21-4358-a2a8-2644ffca19e3" containerName="maas-api" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806436 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29a9d25-cb21-4358-a2a8-2644ffca19e3" containerName="maas-api" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806517 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29a9d25-cb21-4358-a2a8-2644ffca19e3" containerName="maas-api" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806523 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="10afad44-9b46-438a-8de0-affbdb0e8405" containerName="payload-pre-processing" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806529 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e89b6e-0836-4a95-adfb-fb359827d28e" containerName="maas-api" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806535 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806541 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806547 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="7697441d-50ec-428c-90ef-1286c6498dc6" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806555 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a14c40-5fed-42f8-b093-f167eefd7bbd" containerName="payload-processing" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806559 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2ca7f1-d050-4b15-917b-e901168b2199" containerName="istio-proxy" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806565 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a5be49-af4b-45cf-9f8d-8fc49eb4ba75" containerName="payload-pre-processing" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806570 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="054abb30-24f5-4968-a793-a285ad5a7d43" containerName="maas-api" Oct 31 01:07:41.806704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.806576 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5a47e4-abdc-40ef-96c8-7777e2bca4b2" containerName="istio-proxy" Oct 31 01:07:41.811717 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.811696 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:41.899218 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.899181 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:41.899393 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.899373 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:41.899464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.899412 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-metrics-certs\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:41.899464 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.899451 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqkm2\" (UniqueName: \"kubernetes.io/projected/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-kube-api-access-rqkm2\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:41.913864 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.913839 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-3853b6-serving-cert\"" Oct 31 01:07:41.918104 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:41.918079 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q"] Oct 31 01:07:42.000522 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.000445 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:42.000718 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.000563 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:42.000718 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.000585 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-metrics-certs\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:42.000718 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.000630 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqkm2\" (UniqueName: \"kubernetes.io/projected/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-kube-api-access-rqkm2\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:42.000718 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:42.000706 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-serving-cert: secret "maas-api-e2e-extmod-a-3853b6-serving-cert" not found Oct 31 01:07:42.000996 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:42.000786 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls podName:0dc2efd1-88d0-4194-a9ac-9e649b242cf7 nodeName:}" failed. No retries permitted until 2026-10-31 01:07:42.500765782 +0000 UTC m=+3894.976124197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls") pod "maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" (UID: "0dc2efd1-88d0-4194-a9ac-9e649b242cf7") : secret "maas-api-e2e-extmod-a-3853b6-serving-cert" not found Oct 31 01:07:42.001136 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.001118 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:42.003175 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.003151 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-metrics-certs\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:42.064782 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.064747 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqkm2\" (UniqueName: \"kubernetes.io/projected/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-kube-api-access-rqkm2\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:42.506993 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.506949 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:42.509678 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.509655 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:42.725778 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:42.725747 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:43.111282 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:43.111257 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q"] Oct 31 01:07:43.112097 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:07:43.112067 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc2efd1_88d0_4194_a9ac_9e649b242cf7.slice/crio-e64b3f44e9802ee34816ef1af70532269c9bb445733b768ec5245ef47d5f0f02 WatchSource:0}: Error finding container e64b3f44e9802ee34816ef1af70532269c9bb445733b768ec5245ef47d5f0f02: Status 404 returned error can't find the container with id e64b3f44e9802ee34816ef1af70532269c9bb445733b768ec5245ef47d5f0f02 Oct 31 01:07:44.092116 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:44.092087 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" event={"ID":"0dc2efd1-88d0-4194-a9ac-9e649b242cf7","Type":"ContainerStarted","Data":"e64b3f44e9802ee34816ef1af70532269c9bb445733b768ec5245ef47d5f0f02"} Oct 31 01:07:45.101801 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:45.101763 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" event={"ID":"0dc2efd1-88d0-4194-a9ac-9e649b242cf7","Type":"ContainerStarted","Data":"a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1"} Oct 31 01:07:45.102227 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:45.101824 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:45.221285 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:45.221214 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" podStartSLOduration=3.313092191 podStartE2EDuration="4.221199189s" totalImagesPullingTime="908.106998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:45.219512494 +0000 UTC m=+3897.694870913" watchObservedRunningTime="2026-10-31 01:07:45.221199189 +0000 UTC m=+3897.696557612" Oct 31 01:07:48.441266 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:48.441239 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 01:07:48.457660 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:48.457634 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 01:07:48.469002 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:48.468979 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 01:07:48.484016 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:48.483992 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 01:07:49.271406 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:49.271362 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" podUID="84e802b0-f574-4e5d-b253-c889fedc1716" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:07:49.290337 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:49.290307 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx"] Oct 31 01:07:49.290552 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:49.290519 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" containerID="cri-o://f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0" gracePeriod=30 Oct 31 01:07:51.114797 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:51.114763 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:07:54.569642 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.569566 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 01:07:54.649912 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.649883 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-credential-socket\") pod \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " Oct 31 01:07:54.650045 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.649921 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-data\") pod \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " Oct 31 01:07:54.650045 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.649970 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istiod-ca-cert\") pod \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " Oct 31 01:07:54.650045 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.649989 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-kube-api-access-5rszt\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-kube-api-access-5rszt\") pod \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " Oct 31 01:07:54.650045 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650011 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-certs\") pod \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " Oct 31 01:07:54.650045 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650038 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-envoy\") pod \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " Oct 31 01:07:54.650302 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650137 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-socket\") pod \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " Oct 31 01:07:54.650302 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650166 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-podinfo\") pod \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " Oct 31 01:07:54.650302 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650177 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-credential-socket" pod "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" (UID: "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:54.650302 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650188 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-data" pod "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" (UID: "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:54.650302 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650217 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-token\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-token\") pod \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\" (UID: \"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5\") " Oct 31 01:07:54.650564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650428 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-certs" pod "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" (UID: "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:54.650564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650499 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istiod-ca-cert" pod "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" (UID: "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:54.650564 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650511 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-socket" pod "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" (UID: "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:54.650794 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650761 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:54.650794 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650790 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:54.650926 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650802 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:54.650926 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650811 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:54.650926 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.650821 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:54.652632 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.652572 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-podinfo" pod "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" (UID: "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:07:54.652727 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.652651 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-token" pod "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" (UID: "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:54.652925 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.652903 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-envoy" pod "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" (UID: "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:54.652990 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.652951 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-kube-api-access-5rszt" pod "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" (UID: "c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5"). InnerVolumeSpecName "kube-api-access-5rszt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:54.751522 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.751489 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rszt\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-kube-api-access-5rszt\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:54.751522 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.751522 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:54.751759 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.751537 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:54.751759 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:54.751583 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:07:55.179146 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.179111 4969 generic.go:356] "Generic (PLEG): container finished" podID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerID="f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0" exitCode=0 Oct 31 01:07:55.179328 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.179158 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" event={"ID":"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5","Type":"ContainerDied","Data":"f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0"} Oct 31 01:07:55.179328 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.179162 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" Oct 31 01:07:55.179328 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.179205 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx" event={"ID":"c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5","Type":"ContainerDied","Data":"5242ba1ecc859c884f139bcef2a9445dabd1113a59511f7c2d5ab696f7aefd0c"} Oct 31 01:07:55.179328 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.179229 4969 scope.go:122] "RemoveContainer" containerID="f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0" Oct 31 01:07:55.191670 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.191652 4969 scope.go:122] "RemoveContainer" containerID="6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed" Oct 31 01:07:55.201661 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.201639 4969 scope.go:122] "RemoveContainer" containerID="f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0" Oct 31 01:07:55.201906 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:55.201891 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0\": container with ID starting with f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0 not found: ID does not exist" containerID="f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0" Oct 31 01:07:55.201953 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.201913 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0"} err="failed to get container status \"f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0\": rpc error: code = NotFound desc = could not find container \"f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0\": container with ID starting with f6eb78cb1daead5e2a8ee99b6269476423b1ec8c8590df72d05eac36c8c73fa0 not found: ID does not exist" Oct 31 01:07:55.201953 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.201931 4969 scope.go:122] "RemoveContainer" containerID="6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed" Oct 31 01:07:55.202170 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:07:55.202149 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed\": container with ID starting with 6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed not found: ID does not exist" containerID="6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed" Oct 31 01:07:55.202219 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.202176 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed"} err="failed to get container status \"6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed\": rpc error: code = NotFound desc = could not find container \"6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed\": container with ID starting with 6ef6a8a920d773be3af9b53488a0c3a9ed36be375155819b4990fbc085a1c2ed not found: ID does not exist" Oct 31 01:07:55.346081 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.346050 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx"] Oct 31 01:07:55.396254 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.396223 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a7eb09-openshift-default-6f7fdc5845-jmdxx"] Oct 31 01:07:55.738423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.738391 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" path="/var/lib/kubelet/pods/c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5/volumes" Oct 31 01:07:55.767228 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.767193 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g"] Oct 31 01:07:55.767482 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:55.767443 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" containerID="cri-o://4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64" gracePeriod=30 Oct 31 01:07:59.260351 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:07:59.260261 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" podUID="84e802b0-f574-4e5d-b253-c889fedc1716" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:08:00.693348 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:00.693288 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 01:08:00.822018 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:00.821985 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e802b0-f574-4e5d-b253-c889fedc1716-kube-api-access-rsh76\" (UniqueName: \"kubernetes.io/projected/84e802b0-f574-4e5d-b253-c889fedc1716-kube-api-access-rsh76\") pod \"84e802b0-f574-4e5d-b253-c889fedc1716\" (UID: \"84e802b0-f574-4e5d-b253-c889fedc1716\") " Oct 31 01:08:00.822317 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:00.822287 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e802b0-f574-4e5d-b253-c889fedc1716-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e802b0-f574-4e5d-b253-c889fedc1716-plugins-config-volume\") pod \"84e802b0-f574-4e5d-b253-c889fedc1716\" (UID: \"84e802b0-f574-4e5d-b253-c889fedc1716\") " Oct 31 01:08:00.822870 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:00.822845 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e802b0-f574-4e5d-b253-c889fedc1716-plugins-config-volume" pod "84e802b0-f574-4e5d-b253-c889fedc1716" (UID: "84e802b0-f574-4e5d-b253-c889fedc1716"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:00.824909 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:00.824877 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e802b0-f574-4e5d-b253-c889fedc1716-kube-api-access-rsh76" pod "84e802b0-f574-4e5d-b253-c889fedc1716" (UID: "84e802b0-f574-4e5d-b253-c889fedc1716"). InnerVolumeSpecName "kube-api-access-rsh76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:00.923662 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:00.923615 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e802b0-f574-4e5d-b253-c889fedc1716-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:00.923662 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:00.923662 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsh76\" (UniqueName: \"kubernetes.io/projected/84e802b0-f574-4e5d-b253-c889fedc1716-kube-api-access-rsh76\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.018885 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.018851 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 01:08:01.126253 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126224 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-certs\") pod \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " Oct 31 01:08:01.126443 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126332 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-data\") pod \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " Oct 31 01:08:01.126443 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126372 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-token\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-token\") pod \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " Oct 31 01:08:01.126443 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126406 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-kube-api-access-crgbg\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-kube-api-access-crgbg\") pod \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " Oct 31 01:08:01.126443 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126435 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-socket\") pod \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " Oct 31 01:08:01.126888 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126524 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-certs" pod "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" (UID: "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:01.126888 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126581 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istiod-ca-cert\") pod \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " Oct 31 01:08:01.126888 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126697 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-envoy\") pod \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " Oct 31 01:08:01.126888 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126727 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-socket" pod "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" (UID: "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:01.126888 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126727 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-data" pod "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" (UID: "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:01.126888 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126769 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-credential-socket\") pod \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " Oct 31 01:08:01.126888 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.126836 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-podinfo\") pod \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\" (UID: \"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0\") " Oct 31 01:08:01.127336 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.127062 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-credential-socket" pod "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" (UID: "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:01.127511 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.127451 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.127511 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.127481 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.127511 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.127498 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.127511 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.127514 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.127511 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.127511 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istiod-ca-cert" pod "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" (UID: "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:01.129039 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.129002 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-kube-api-access-crgbg" pod "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" (UID: "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0"). InnerVolumeSpecName "kube-api-access-crgbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:01.129039 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.129022 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-podinfo" pod "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" (UID: "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:08:01.129206 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.129138 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-token" pod "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" (UID: "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:01.129206 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.129182 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-envoy" pod "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" (UID: "4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:01.226519 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.226486 4969 generic.go:356] "Generic (PLEG): container finished" podID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerID="4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64" exitCode=0 Oct 31 01:08:01.226728 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.226536 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" Oct 31 01:08:01.226728 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.226575 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" event={"ID":"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0","Type":"ContainerDied","Data":"4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64"} Oct 31 01:08:01.226728 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.226655 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g" event={"ID":"4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0","Type":"ContainerDied","Data":"2e8585d37a9b04143861799674e3e90f3b5d6ba5ec1246621ea3e48c38dafda9"} Oct 31 01:08:01.226728 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.226690 4969 scope.go:122] "RemoveContainer" containerID="4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64" Oct 31 01:08:01.228221 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.228197 4969 generic.go:356] "Generic (PLEG): container finished" podID="84e802b0-f574-4e5d-b253-c889fedc1716" containerID="b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807" exitCode=0 Oct 31 01:08:01.228374 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.228240 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" Oct 31 01:08:01.228374 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.228308 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.228374 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.228332 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.228374 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.228280 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" event={"ID":"84e802b0-f574-4e5d-b253-c889fedc1716","Type":"ContainerDied","Data":"b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807"} Oct 31 01:08:01.228618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.228409 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8" event={"ID":"84e802b0-f574-4e5d-b253-c889fedc1716","Type":"ContainerDied","Data":"d14694cce8cab3cea4a4b7b441a338c34293bf8ed7f314ed44ba2f2dabd9357e"} Oct 31 01:08:01.228618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.228531 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.228779 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.228759 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.228828 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.228789 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crgbg\" (UniqueName: \"kubernetes.io/projected/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0-kube-api-access-crgbg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:01.249056 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.249031 4969 scope.go:122] "RemoveContainer" containerID="efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845" Oct 31 01:08:01.260185 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.260163 4969 scope.go:122] "RemoveContainer" containerID="4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64" Oct 31 01:08:01.260504 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:08:01.260475 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64\": container with ID starting with 4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64 not found: ID does not exist" containerID="4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64" Oct 31 01:08:01.260615 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.260519 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64"} err="failed to get container status \"4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64\": rpc error: code = NotFound desc = could not find container \"4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64\": container with ID starting with 4d5fc444fc2324c13b8ba9b39238937825cff17f1b4b6db0254cf14966c3cc64 not found: ID does not exist" Oct 31 01:08:01.260615 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.260545 4969 scope.go:122] "RemoveContainer" containerID="efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845" Oct 31 01:08:01.260902 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:08:01.260880 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845\": container with ID starting with efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845 not found: ID does not exist" containerID="efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845" Oct 31 01:08:01.260979 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.260911 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845"} err="failed to get container status \"efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845\": rpc error: code = NotFound desc = could not find container \"efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845\": container with ID starting with efb05d40157c3e185d3ce35ba5e8c7ab28a20e54d4135096cc5760edb59e6845 not found: ID does not exist" Oct 31 01:08:01.260979 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.260932 4969 scope.go:122] "RemoveContainer" containerID="b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807" Oct 31 01:08:01.270973 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.270947 4969 scope.go:122] "RemoveContainer" containerID="b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807" Oct 31 01:08:01.271267 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:08:01.271249 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807\": container with ID starting with b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807 not found: ID does not exist" containerID="b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807" Oct 31 01:08:01.271342 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.271276 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807"} err="failed to get container status \"b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807\": rpc error: code = NotFound desc = could not find container \"b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807\": container with ID starting with b25e2f34bfa29d39e4ed161e876b5e2885c38b19956107db423a8f6e65a6f807 not found: ID does not exist" Oct 31 01:08:01.393328 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.393293 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g"] Oct 31 01:08:01.399578 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.399545 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f13553-openshift-default-55679f5995-zz98g"] Oct 31 01:08:01.509428 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.509381 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8"] Oct 31 01:08:01.512651 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.512574 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bf325c-5bbfb6885-5dww8"] Oct 31 01:08:01.660110 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.660077 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674"] Oct 31 01:08:01.734702 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.734669 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" path="/var/lib/kubelet/pods/4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0/volumes" Oct 31 01:08:01.735128 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:01.735113 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e802b0-f574-4e5d-b253-c889fedc1716" path="/var/lib/kubelet/pods/84e802b0-f574-4e5d-b253-c889fedc1716/volumes" Oct 31 01:08:02.728513 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:08:02.728479 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:08:06.660988 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:06.660948 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" podUID="36e09f55-204e-430c-8cc7-47fb2356889a" containerName="payload-processing" containerID="cri-o://37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9" gracePeriod=25 Oct 31 01:08:06.938018 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:06.937990 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:08:06.993696 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:06.993667 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e09f55-204e-430c-8cc7-47fb2356889a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e09f55-204e-430c-8cc7-47fb2356889a-plugins-config-volume\") pod \"36e09f55-204e-430c-8cc7-47fb2356889a\" (UID: \"36e09f55-204e-430c-8cc7-47fb2356889a\") " Oct 31 01:08:06.993829 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:06.993765 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e09f55-204e-430c-8cc7-47fb2356889a-kube-api-access-f2dhx\" (UniqueName: \"kubernetes.io/projected/36e09f55-204e-430c-8cc7-47fb2356889a-kube-api-access-f2dhx\") pod \"36e09f55-204e-430c-8cc7-47fb2356889a\" (UID: \"36e09f55-204e-430c-8cc7-47fb2356889a\") " Oct 31 01:08:06.994022 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:06.994000 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e09f55-204e-430c-8cc7-47fb2356889a-plugins-config-volume" pod "36e09f55-204e-430c-8cc7-47fb2356889a" (UID: "36e09f55-204e-430c-8cc7-47fb2356889a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:06.996097 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:06.996076 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e09f55-204e-430c-8cc7-47fb2356889a-kube-api-access-f2dhx" pod "36e09f55-204e-430c-8cc7-47fb2356889a" (UID: "36e09f55-204e-430c-8cc7-47fb2356889a"). InnerVolumeSpecName "kube-api-access-f2dhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:07.094869 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.094835 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e09f55-204e-430c-8cc7-47fb2356889a-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:07.094869 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.094862 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2dhx\" (UniqueName: \"kubernetes.io/projected/36e09f55-204e-430c-8cc7-47fb2356889a-kube-api-access-f2dhx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:07.276826 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.276740 4969 generic.go:356] "Generic (PLEG): container finished" podID="36e09f55-204e-430c-8cc7-47fb2356889a" containerID="37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9" exitCode=0 Oct 31 01:08:07.276826 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.276786 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" Oct 31 01:08:07.277046 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.276816 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" event={"ID":"36e09f55-204e-430c-8cc7-47fb2356889a","Type":"ContainerDied","Data":"37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9"} Oct 31 01:08:07.277046 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.276886 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674" event={"ID":"36e09f55-204e-430c-8cc7-47fb2356889a","Type":"ContainerDied","Data":"a382245431d999d95e5ed1fb8f160886f0f431d06cb79a3154f000de15004bcc"} Oct 31 01:08:07.277046 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.276915 4969 scope.go:122] "RemoveContainer" containerID="37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9" Oct 31 01:08:07.288449 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.288431 4969 scope.go:122] "RemoveContainer" containerID="37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9" Oct 31 01:08:07.288706 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:08:07.288685 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9\": container with ID starting with 37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9 not found: ID does not exist" containerID="37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9" Oct 31 01:08:07.288784 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.288712 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9"} err="failed to get container status \"37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9\": rpc error: code = NotFound desc = could not find container \"37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9\": container with ID starting with 37cdff8a4e90981812d3885467c5d62d4e49ca2fb99471c7b0aee98867b0e9d9 not found: ID does not exist" Oct 31 01:08:07.442303 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.442265 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674"] Oct 31 01:08:07.491965 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.491929 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e8808d-74fc6df98c-x4674"] Oct 31 01:08:07.734639 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.734575 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e09f55-204e-430c-8cc7-47fb2356889a" path="/var/lib/kubelet/pods/36e09f55-204e-430c-8cc7-47fb2356889a/volumes" Oct 31 01:08:07.895970 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.895933 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs"] Oct 31 01:08:07.896194 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:07.896168 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" podUID="4234bd78-5607-40a7-8f67-94848b2cf4be" containerName="maas-api" containerID="cri-o://5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325" gracePeriod=30 Oct 31 01:08:08.171816 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.171670 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:08:08.207830 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.207798 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-metrics-certs\") pod \"4234bd78-5607-40a7-8f67-94848b2cf4be\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " Oct 31 01:08:08.207981 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.207899 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4234bd78-5607-40a7-8f67-94848b2cf4be-kube-api-access-6h8gx\" (UniqueName: \"kubernetes.io/projected/4234bd78-5607-40a7-8f67-94848b2cf4be-kube-api-access-6h8gx\") pod \"4234bd78-5607-40a7-8f67-94848b2cf4be\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " Oct 31 01:08:08.207981 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.207951 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls\") pod \"4234bd78-5607-40a7-8f67-94848b2cf4be\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " Oct 31 01:08:08.207981 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.207969 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4234bd78-5607-40a7-8f67-94848b2cf4be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4234bd78-5607-40a7-8f67-94848b2cf4be-openshift-service-ca\") pod \"4234bd78-5607-40a7-8f67-94848b2cf4be\" (UID: \"4234bd78-5607-40a7-8f67-94848b2cf4be\") " Oct 31 01:08:08.208422 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.208392 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4234bd78-5607-40a7-8f67-94848b2cf4be-openshift-service-ca" pod "4234bd78-5607-40a7-8f67-94848b2cf4be" (UID: "4234bd78-5607-40a7-8f67-94848b2cf4be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:08.210200 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.210170 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-metrics-certs" pod "4234bd78-5607-40a7-8f67-94848b2cf4be" (UID: "4234bd78-5607-40a7-8f67-94848b2cf4be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:08.210331 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.210214 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls" pod "4234bd78-5607-40a7-8f67-94848b2cf4be" (UID: "4234bd78-5607-40a7-8f67-94848b2cf4be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:08.210331 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.210270 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4234bd78-5607-40a7-8f67-94848b2cf4be-kube-api-access-6h8gx" pod "4234bd78-5607-40a7-8f67-94848b2cf4be" (UID: "4234bd78-5607-40a7-8f67-94848b2cf4be"). InnerVolumeSpecName "kube-api-access-6h8gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:08.286744 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.286661 4969 generic.go:356] "Generic (PLEG): container finished" podID="4234bd78-5607-40a7-8f67-94848b2cf4be" containerID="5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325" exitCode=0 Oct 31 01:08:08.286744 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.286729 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" Oct 31 01:08:08.286968 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.286757 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" event={"ID":"4234bd78-5607-40a7-8f67-94848b2cf4be","Type":"ContainerDied","Data":"5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325"} Oct 31 01:08:08.286968 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.286824 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs" event={"ID":"4234bd78-5607-40a7-8f67-94848b2cf4be","Type":"ContainerDied","Data":"e8779d6196fe7eab2bdab9ebf72ce8dc26133c6de3d4cb14e46c378167695602"} Oct 31 01:08:08.286968 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.286892 4969 scope.go:122] "RemoveContainer" containerID="5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325" Oct 31 01:08:08.301492 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.301469 4969 scope.go:122] "RemoveContainer" containerID="5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325" Oct 31 01:08:08.301836 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:08:08.301807 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325\": container with ID starting with 5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325 not found: ID does not exist" containerID="5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325" Oct 31 01:08:08.301980 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.301847 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325"} err="failed to get container status \"5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325\": rpc error: code = NotFound desc = could not find container \"5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325\": container with ID starting with 5ff4f4a781a7e6837be0e97d54f6438bdc831dcf35c96ba92ab1a96951b51325 not found: ID does not exist" Oct 31 01:08:08.309169 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.309145 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h8gx\" (UniqueName: \"kubernetes.io/projected/4234bd78-5607-40a7-8f67-94848b2cf4be-kube-api-access-6h8gx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:08.309169 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.309169 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:08.309335 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.309179 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4234bd78-5607-40a7-8f67-94848b2cf4be-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:08.309335 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.309188 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4234bd78-5607-40a7-8f67-94848b2cf4be-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:08.455136 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.455101 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs"] Oct 31 01:08:08.460347 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:08.460322 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aaa842-dd58f49fb-lfjhs"] Oct 31 01:08:09.734296 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:09.734259 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4234bd78-5607-40a7-8f67-94848b2cf4be" path="/var/lib/kubelet/pods/4234bd78-5607-40a7-8f67-94848b2cf4be/volumes" Oct 31 01:08:14.176791 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.176751 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727"] Oct 31 01:08:14.177413 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.177105 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" podUID="1558a8d2-5076-43b8-8817-fdead1d9ada7" containerName="maas-api" containerID="cri-o://5c73757958c0b82200dc1aee5438bdaf5bac95b56863390a3758431e3726de35" gracePeriod=30 Oct 31 01:08:14.332668 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.332636 4969 generic.go:356] "Generic (PLEG): container finished" podID="1558a8d2-5076-43b8-8817-fdead1d9ada7" containerID="5c73757958c0b82200dc1aee5438bdaf5bac95b56863390a3758431e3726de35" exitCode=0 Oct 31 01:08:14.332876 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.332720 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" event={"ID":"1558a8d2-5076-43b8-8817-fdead1d9ada7","Type":"ContainerDied","Data":"5c73757958c0b82200dc1aee5438bdaf5bac95b56863390a3758431e3726de35"} Oct 31 01:08:14.461267 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.461240 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 01:08:14.584161 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.584127 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1558a8d2-5076-43b8-8817-fdead1d9ada7-kube-api-access-2kfqd\" (UniqueName: \"kubernetes.io/projected/1558a8d2-5076-43b8-8817-fdead1d9ada7-kube-api-access-2kfqd\") pod \"1558a8d2-5076-43b8-8817-fdead1d9ada7\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " Oct 31 01:08:14.584417 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.584240 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls\") pod \"1558a8d2-5076-43b8-8817-fdead1d9ada7\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " Oct 31 01:08:14.584417 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.584313 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1558a8d2-5076-43b8-8817-fdead1d9ada7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1558a8d2-5076-43b8-8817-fdead1d9ada7-openshift-service-ca\") pod \"1558a8d2-5076-43b8-8817-fdead1d9ada7\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " Oct 31 01:08:14.584653 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.584457 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-metrics-certs\") pod \"1558a8d2-5076-43b8-8817-fdead1d9ada7\" (UID: \"1558a8d2-5076-43b8-8817-fdead1d9ada7\") " Oct 31 01:08:14.585710 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.585659 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1558a8d2-5076-43b8-8817-fdead1d9ada7-openshift-service-ca" pod "1558a8d2-5076-43b8-8817-fdead1d9ada7" (UID: "1558a8d2-5076-43b8-8817-fdead1d9ada7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:14.591634 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.591570 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1558a8d2-5076-43b8-8817-fdead1d9ada7-kube-api-access-2kfqd" pod "1558a8d2-5076-43b8-8817-fdead1d9ada7" (UID: "1558a8d2-5076-43b8-8817-fdead1d9ada7"). InnerVolumeSpecName "kube-api-access-2kfqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:14.591764 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.591710 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls" pod "1558a8d2-5076-43b8-8817-fdead1d9ada7" (UID: "1558a8d2-5076-43b8-8817-fdead1d9ada7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:14.592033 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.592004 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-metrics-certs" pod "1558a8d2-5076-43b8-8817-fdead1d9ada7" (UID: "1558a8d2-5076-43b8-8817-fdead1d9ada7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:14.686041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.685967 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1558a8d2-5076-43b8-8817-fdead1d9ada7-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:14.686041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.685996 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:14.686041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.686005 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kfqd\" (UniqueName: \"kubernetes.io/projected/1558a8d2-5076-43b8-8817-fdead1d9ada7-kube-api-access-2kfqd\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:14.686041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:14.686015 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1558a8d2-5076-43b8-8817-fdead1d9ada7-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:15.341543 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:15.341512 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" Oct 31 01:08:15.342039 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:15.341554 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727" event={"ID":"1558a8d2-5076-43b8-8817-fdead1d9ada7","Type":"ContainerDied","Data":"7694f109d94ef86a70b604ecc396e23fb1355994010aed7514f877313852a53b"} Oct 31 01:08:15.342039 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:15.341646 4969 scope.go:122] "RemoveContainer" containerID="5c73757958c0b82200dc1aee5438bdaf5bac95b56863390a3758431e3726de35" Oct 31 01:08:15.506376 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:15.506264 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727"] Oct 31 01:08:15.555949 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:15.555912 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4e1a6c-7db6578695-lx727"] Oct 31 01:08:15.734805 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:15.734775 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1558a8d2-5076-43b8-8817-fdead1d9ada7" path="/var/lib/kubelet/pods/1558a8d2-5076-43b8-8817-fdead1d9ada7/volumes" Oct 31 01:08:28.721132 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:28.721097 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c"] Oct 31 01:08:29.132634 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:29.132602 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6"] Oct 31 01:08:32.729583 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:08:32.729542 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:08:34.133843 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.133778 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" podUID="bee68c13-b7f1-4612-a57b-3b928d9d0019" containerName="payload-processing" containerID="cri-o://489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018" gracePeriod=25 Oct 31 01:08:34.395201 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.395134 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:08:34.483432 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.483400 4969 generic.go:356] "Generic (PLEG): container finished" podID="bee68c13-b7f1-4612-a57b-3b928d9d0019" containerID="489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018" exitCode=0 Oct 31 01:08:34.483619 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.483436 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" Oct 31 01:08:34.483619 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.483471 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" event={"ID":"bee68c13-b7f1-4612-a57b-3b928d9d0019","Type":"ContainerDied","Data":"489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018"} Oct 31 01:08:34.483619 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.483516 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6" event={"ID":"bee68c13-b7f1-4612-a57b-3b928d9d0019","Type":"ContainerDied","Data":"8d154efa2a663298c24d178f66efa5253a84e0c87cfdfdc582424deacdb5a5aa"} Oct 31 01:08:34.483619 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.483541 4969 scope.go:122] "RemoveContainer" containerID="489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018" Oct 31 01:08:34.493900 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.493875 4969 scope.go:122] "RemoveContainer" containerID="489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018" Oct 31 01:08:34.494168 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:08:34.494146 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018\": container with ID starting with 489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018 not found: ID does not exist" containerID="489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018" Oct 31 01:08:34.494237 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.494178 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018"} err="failed to get container status \"489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018\": rpc error: code = NotFound desc = could not find container \"489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018\": container with ID starting with 489ab394056163ee1a5f0430982f48275b2eeb08747a4f691b7efc5caaa8e018 not found: ID does not exist" Oct 31 01:08:34.504617 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.504578 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee68c13-b7f1-4612-a57b-3b928d9d0019-kube-api-access-2bwkj\" (UniqueName: \"kubernetes.io/projected/bee68c13-b7f1-4612-a57b-3b928d9d0019-kube-api-access-2bwkj\") pod \"bee68c13-b7f1-4612-a57b-3b928d9d0019\" (UID: \"bee68c13-b7f1-4612-a57b-3b928d9d0019\") " Oct 31 01:08:34.504691 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.504652 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee68c13-b7f1-4612-a57b-3b928d9d0019-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee68c13-b7f1-4612-a57b-3b928d9d0019-plugins-config-volume\") pod \"bee68c13-b7f1-4612-a57b-3b928d9d0019\" (UID: \"bee68c13-b7f1-4612-a57b-3b928d9d0019\") " Oct 31 01:08:34.504996 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.504976 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee68c13-b7f1-4612-a57b-3b928d9d0019-plugins-config-volume" pod "bee68c13-b7f1-4612-a57b-3b928d9d0019" (UID: "bee68c13-b7f1-4612-a57b-3b928d9d0019"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:34.505150 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.505131 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee68c13-b7f1-4612-a57b-3b928d9d0019-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:34.506768 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.506748 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee68c13-b7f1-4612-a57b-3b928d9d0019-kube-api-access-2bwkj" pod "bee68c13-b7f1-4612-a57b-3b928d9d0019" (UID: "bee68c13-b7f1-4612-a57b-3b928d9d0019"). InnerVolumeSpecName "kube-api-access-2bwkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:34.606191 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.606157 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bwkj\" (UniqueName: \"kubernetes.io/projected/bee68c13-b7f1-4612-a57b-3b928d9d0019-kube-api-access-2bwkj\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:34.913194 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.913157 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6"] Oct 31 01:08:34.962353 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:34.962322 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-395b06-699bc5d4b7-n5xn6"] Oct 31 01:08:35.734661 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:35.734619 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee68c13-b7f1-4612-a57b-3b928d9d0019" path="/var/lib/kubelet/pods/bee68c13-b7f1-4612-a57b-3b928d9d0019/volumes" Oct 31 01:08:35.748649 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:35.748619 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg"] Oct 31 01:08:35.857690 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:35.857656 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np"] Oct 31 01:08:40.749215 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:40.749175 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" podUID="59809d8f-4894-49ce-ab6c-361e00e41285" containerName="payload-processing" containerID="cri-o://dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7" gracePeriod=25 Oct 31 01:08:40.858859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:40.858822 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" podUID="3487e6ad-e165-46d6-942d-f9c25c800ba5" containerName="payload-pre-processing" containerID="cri-o://a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851" gracePeriod=25 Oct 31 01:08:42.045084 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:42.045042 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" podUID="3487e6ad-e165-46d6-942d-f9c25c800ba5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:08:42.045559 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:42.045042 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" podUID="59809d8f-4894-49ce-ab6c-361e00e41285" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:08:43.742492 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:43.742443 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" containerID="cri-o://143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54" gracePeriod=45 Oct 31 01:08:44.010861 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.010803 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 01:08:44.109713 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.109681 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3b5687-3b27-4a37-a867-73a2d0bec08a-kube-api-access-fxs5x\" (UniqueName: \"kubernetes.io/projected/ff3b5687-3b27-4a37-a867-73a2d0bec08a-kube-api-access-fxs5x\") pod \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " Oct 31 01:08:44.109880 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.109721 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-dshm\") pod \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " Oct 31 01:08:44.109880 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.109777 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-home\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-home\") pod \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " Oct 31 01:08:44.109880 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.109807 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tmp-dir\") pod \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " Oct 31 01:08:44.109880 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.109845 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tls-certs\") pod \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " Oct 31 01:08:44.110115 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.109894 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-model-cache\") pod \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\" (UID: \"ff3b5687-3b27-4a37-a867-73a2d0bec08a\") " Oct 31 01:08:44.110115 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.110070 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tmp-dir" pod "ff3b5687-3b27-4a37-a867-73a2d0bec08a" (UID: "ff3b5687-3b27-4a37-a867-73a2d0bec08a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.110115 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.110082 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-home" pod "ff3b5687-3b27-4a37-a867-73a2d0bec08a" (UID: "ff3b5687-3b27-4a37-a867-73a2d0bec08a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.110293 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.110255 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-model-cache" pod "ff3b5687-3b27-4a37-a867-73a2d0bec08a" (UID: "ff3b5687-3b27-4a37-a867-73a2d0bec08a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.110414 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.110370 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.110608 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.110482 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.110608 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.110569 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.112245 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.112217 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tls-certs" pod "ff3b5687-3b27-4a37-a867-73a2d0bec08a" (UID: "ff3b5687-3b27-4a37-a867-73a2d0bec08a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:44.112358 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.112220 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-dshm" pod "ff3b5687-3b27-4a37-a867-73a2d0bec08a" (UID: "ff3b5687-3b27-4a37-a867-73a2d0bec08a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:08:44.112358 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.112299 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3b5687-3b27-4a37-a867-73a2d0bec08a-kube-api-access-fxs5x" pod "ff3b5687-3b27-4a37-a867-73a2d0bec08a" (UID: "ff3b5687-3b27-4a37-a867-73a2d0bec08a"). InnerVolumeSpecName "kube-api-access-fxs5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:44.212147 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.212112 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3b5687-3b27-4a37-a867-73a2d0bec08a-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.212147 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.212142 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxs5x\" (UniqueName: \"kubernetes.io/projected/ff3b5687-3b27-4a37-a867-73a2d0bec08a-kube-api-access-fxs5x\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.212147 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.212153 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3b5687-3b27-4a37-a867-73a2d0bec08a-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.559224 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.559192 4969 generic.go:356] "Generic (PLEG): container finished" podID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerID="143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54" exitCode=0 Oct 31 01:08:44.559417 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.559235 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" Oct 31 01:08:44.559417 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.559273 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" event={"ID":"ff3b5687-3b27-4a37-a867-73a2d0bec08a","Type":"ContainerDied","Data":"143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54"} Oct 31 01:08:44.559417 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.559320 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c" event={"ID":"ff3b5687-3b27-4a37-a867-73a2d0bec08a","Type":"ContainerDied","Data":"1e32a94422a98c82e0a9615164a4d92a223d80ab8463be24ed061a55acc48eaf"} Oct 31 01:08:44.559417 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.559339 4969 scope.go:122] "RemoveContainer" containerID="143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54" Oct 31 01:08:44.570883 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.570868 4969 scope.go:122] "RemoveContainer" containerID="143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54" Oct 31 01:08:44.571154 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:08:44.571134 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54\": container with ID starting with 143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54 not found: ID does not exist" containerID="143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54" Oct 31 01:08:44.571227 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.571167 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54"} err="failed to get container status \"143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54\": rpc error: code = NotFound desc = could not find container \"143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54\": container with ID starting with 143d412e82d732b3729051f75d768114a5e4a2e89cbfdb92600097ee79e17a54 not found: ID does not exist" Oct 31 01:08:44.727137 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.727108 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c"] Oct 31 01:08:44.778382 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:44.778310 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7abb95/distinct-w3-7abb95-kserve-589b4c6b8d-8q45c"] Oct 31 01:08:45.735401 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:45.735369 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" path="/var/lib/kubelet/pods/ff3b5687-3b27-4a37-a867-73a2d0bec08a/volumes" Oct 31 01:08:52.044831 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:52.044796 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" podUID="3487e6ad-e165-46d6-942d-f9c25c800ba5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:08:52.044831 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:52.044807 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" podUID="59809d8f-4894-49ce-ab6c-361e00e41285" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:08:55.576271 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:55.576235 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x"] Oct 31 01:08:55.625961 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:55.625916 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26"] Oct 31 01:08:55.629022 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:08:55.628987 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk"] Oct 31 01:09:01.032850 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.032803 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 01:09:01.129931 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.129906 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 01:09:01.195917 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.195878 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59809d8f-4894-49ce-ab6c-361e00e41285-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59809d8f-4894-49ce-ab6c-361e00e41285-plugins-config-volume\") pod \"59809d8f-4894-49ce-ab6c-361e00e41285\" (UID: \"59809d8f-4894-49ce-ab6c-361e00e41285\") " Oct 31 01:09:01.195917 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.195922 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59809d8f-4894-49ce-ab6c-361e00e41285-kube-api-access-8m2sf\" (UniqueName: \"kubernetes.io/projected/59809d8f-4894-49ce-ab6c-361e00e41285-kube-api-access-8m2sf\") pod \"59809d8f-4894-49ce-ab6c-361e00e41285\" (UID: \"59809d8f-4894-49ce-ab6c-361e00e41285\") " Oct 31 01:09:01.196329 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.196304 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59809d8f-4894-49ce-ab6c-361e00e41285-plugins-config-volume" pod "59809d8f-4894-49ce-ab6c-361e00e41285" (UID: "59809d8f-4894-49ce-ab6c-361e00e41285"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:01.198363 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.198336 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59809d8f-4894-49ce-ab6c-361e00e41285-kube-api-access-8m2sf" pod "59809d8f-4894-49ce-ab6c-361e00e41285" (UID: "59809d8f-4894-49ce-ab6c-361e00e41285"). InnerVolumeSpecName "kube-api-access-8m2sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:01.297510 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.297413 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3487e6ad-e165-46d6-942d-f9c25c800ba5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3487e6ad-e165-46d6-942d-f9c25c800ba5-plugins-config-volume\") pod \"3487e6ad-e165-46d6-942d-f9c25c800ba5\" (UID: \"3487e6ad-e165-46d6-942d-f9c25c800ba5\") " Oct 31 01:09:01.297708 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.297513 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3487e6ad-e165-46d6-942d-f9c25c800ba5-kube-api-access-p8p27\" (UniqueName: \"kubernetes.io/projected/3487e6ad-e165-46d6-942d-f9c25c800ba5-kube-api-access-p8p27\") pod \"3487e6ad-e165-46d6-942d-f9c25c800ba5\" (UID: \"3487e6ad-e165-46d6-942d-f9c25c800ba5\") " Oct 31 01:09:01.297894 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.297872 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3487e6ad-e165-46d6-942d-f9c25c800ba5-plugins-config-volume" pod "3487e6ad-e165-46d6-942d-f9c25c800ba5" (UID: "3487e6ad-e165-46d6-942d-f9c25c800ba5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:01.298141 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.298116 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3487e6ad-e165-46d6-942d-f9c25c800ba5-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:01.298386 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.298147 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59809d8f-4894-49ce-ab6c-361e00e41285-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:01.298386 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.298162 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m2sf\" (UniqueName: \"kubernetes.io/projected/59809d8f-4894-49ce-ab6c-361e00e41285-kube-api-access-8m2sf\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:01.299992 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.299965 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3487e6ad-e165-46d6-942d-f9c25c800ba5-kube-api-access-p8p27" pod "3487e6ad-e165-46d6-942d-f9c25c800ba5" (UID: "3487e6ad-e165-46d6-942d-f9c25c800ba5"). InnerVolumeSpecName "kube-api-access-p8p27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:01.399772 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.399737 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8p27\" (UniqueName: \"kubernetes.io/projected/3487e6ad-e165-46d6-942d-f9c25c800ba5-kube-api-access-p8p27\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:01.509713 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.509668 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7"] Oct 31 01:09:01.510300 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510284 4969 cpu_manager.go:414] "containerMap: removing container" podUID="3487e6ad-e165-46d6-942d-f9c25c800ba5" containerName="payload-pre-processing" Oct 31 01:09:01.510364 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510300 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3487e6ad-e165-46d6-942d-f9c25c800ba5" containerName="payload-pre-processing" Oct 31 01:09:01.510364 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510329 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" Oct 31 01:09:01.510364 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510335 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" Oct 31 01:09:01.510364 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510342 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4234bd78-5607-40a7-8f67-94848b2cf4be" containerName="maas-api" Oct 31 01:09:01.510364 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510347 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4234bd78-5607-40a7-8f67-94848b2cf4be" containerName="maas-api" Oct 31 01:09:01.510364 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510359 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" Oct 31 01:09:01.510364 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510363 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510374 4969 cpu_manager.go:414] "containerMap: removing container" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510379 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510386 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510390 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510404 4969 cpu_manager.go:414] "containerMap: removing container" podUID="1558a8d2-5076-43b8-8817-fdead1d9ada7" containerName="maas-api" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510410 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1558a8d2-5076-43b8-8817-fdead1d9ada7" containerName="maas-api" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510418 4969 cpu_manager.go:414] "containerMap: removing container" podUID="36e09f55-204e-430c-8cc7-47fb2356889a" containerName="payload-processing" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510424 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e09f55-204e-430c-8cc7-47fb2356889a" containerName="payload-processing" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510430 4969 cpu_manager.go:414] "containerMap: removing container" podUID="59809d8f-4894-49ce-ab6c-361e00e41285" containerName="payload-processing" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510434 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59809d8f-4894-49ce-ab6c-361e00e41285" containerName="payload-processing" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510441 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510445 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510452 4969 cpu_manager.go:414] "containerMap: removing container" podUID="bee68c13-b7f1-4612-a57b-3b928d9d0019" containerName="payload-processing" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510456 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee68c13-b7f1-4612-a57b-3b928d9d0019" containerName="payload-processing" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510465 4969 cpu_manager.go:414] "containerMap: removing container" podUID="84e802b0-f574-4e5d-b253-c889fedc1716" containerName="payload-processing" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510470 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e802b0-f574-4e5d-b253-c889fedc1716" containerName="payload-processing" Oct 31 01:09:01.510618 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510613 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510632 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4767fc84-2b65-4ec8-87cd-e7e2f1c2f4e0" containerName="istio-proxy" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510643 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510652 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="1558a8d2-5076-43b8-8817-fdead1d9ada7" containerName="maas-api" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510659 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e09f55-204e-430c-8cc7-47fb2356889a" containerName="payload-processing" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510664 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="59809d8f-4894-49ce-ab6c-361e00e41285" containerName="payload-processing" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510670 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3b5687-3b27-4a37-a867-73a2d0bec08a" containerName="main" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510674 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee68c13-b7f1-4612-a57b-3b928d9d0019" containerName="payload-processing" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510680 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e802b0-f574-4e5d-b253-c889fedc1716" containerName="payload-processing" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510692 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="3487e6ad-e165-46d6-942d-f9c25c800ba5" containerName="payload-pre-processing" Oct 31 01:09:01.511272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.510703 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="4234bd78-5607-40a7-8f67-94848b2cf4be" containerName="maas-api" Oct 31 01:09:01.514132 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.514112 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.579671 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.566911 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d7412d-openshift-default-dockercfg-ptn6k\"" Oct 31 01:09:01.579671 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.579316 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7"] Oct 31 01:09:01.679401 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.679367 4969 generic.go:356] "Generic (PLEG): container finished" podID="3487e6ad-e165-46d6-942d-f9c25c800ba5" containerID="a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851" exitCode=0 Oct 31 01:09:01.679588 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.679416 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" Oct 31 01:09:01.679588 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.679452 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" event={"ID":"3487e6ad-e165-46d6-942d-f9c25c800ba5","Type":"ContainerDied","Data":"a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851"} Oct 31 01:09:01.679588 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.679513 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np" event={"ID":"3487e6ad-e165-46d6-942d-f9c25c800ba5","Type":"ContainerDied","Data":"6e77b2815f6be9296bd9c67861d8e4e709d1884a193e83bc35a1853273360525"} Oct 31 01:09:01.679588 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.679544 4969 scope.go:122] "RemoveContainer" containerID="a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851" Oct 31 01:09:01.681016 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.680988 4969 generic.go:356] "Generic (PLEG): container finished" podID="59809d8f-4894-49ce-ab6c-361e00e41285" containerID="dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7" exitCode=0 Oct 31 01:09:01.681131 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.681042 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" Oct 31 01:09:01.681131 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.681064 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" event={"ID":"59809d8f-4894-49ce-ab6c-361e00e41285","Type":"ContainerDied","Data":"dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7"} Oct 31 01:09:01.681217 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.681127 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg" event={"ID":"59809d8f-4894-49ce-ab6c-361e00e41285","Type":"ContainerDied","Data":"2e28d7f44fc38d54ab7846d45f22829708999f9ed98c8cbb8cccec7321ccae96"} Oct 31 01:09:01.690083 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.690060 4969 scope.go:122] "RemoveContainer" containerID="a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851" Oct 31 01:09:01.690375 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:01.690355 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851\": container with ID starting with a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851 not found: ID does not exist" containerID="a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851" Oct 31 01:09:01.690475 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.690382 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851"} err="failed to get container status \"a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851\": rpc error: code = NotFound desc = could not find container \"a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851\": container with ID starting with a4c96ea3ec35075a395f9704ac9804214975a9673714bf850da705106b2c3851 not found: ID does not exist" Oct 31 01:09:01.690475 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.690399 4969 scope.go:122] "RemoveContainer" containerID="dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7" Oct 31 01:09:01.700673 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.700654 4969 scope.go:122] "RemoveContainer" containerID="dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7" Oct 31 01:09:01.700951 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:01.700931 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7\": container with ID starting with dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7 not found: ID does not exist" containerID="dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7" Oct 31 01:09:01.701006 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.700960 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7"} err="failed to get container status \"dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7\": rpc error: code = NotFound desc = could not find container \"dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7\": container with ID starting with dfc72325613a64a00ce2fe3869a434332a724e79a9a2e3fd992a0e3c1c4fb5a7 not found: ID does not exist" Oct 31 01:09:01.703659 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.703639 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be65041-985f-484f-a025-a2e0c8d4ca6a-istiod-ca-cert\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.703732 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.703675 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-certs\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.703732 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.703705 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-socket\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.703892 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.703870 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-podinfo\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.703929 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.703914 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-data\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.703966 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.703942 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-token\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.704049 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.704036 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-envoy\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.704092 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.704067 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkg78\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-kube-api-access-nkg78\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.704130 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.704096 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-credential-socket\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805187 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805147 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be65041-985f-484f-a025-a2e0c8d4ca6a-istiod-ca-cert\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805363 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805200 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-certs\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805363 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805236 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-socket\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805363 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805279 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-podinfo\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805363 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805298 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-data\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805363 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805352 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-token\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805733 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805446 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-envoy\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805733 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805481 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkg78\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-kube-api-access-nkg78\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805733 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805515 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-credential-socket\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805733 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805722 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-certs\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.805979 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.805950 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-data\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.806048 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.806024 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be65041-985f-484f-a025-a2e0c8d4ca6a-istiod-ca-cert\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.806110 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.806080 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-socket\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.806223 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.806201 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-credential-socket\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.807944 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.807921 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-envoy\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.808059 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.808039 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-podinfo\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.845154 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.845071 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg"] Oct 31 01:09:01.849554 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.849528 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7abb95-7d59cf95fc-bwbfg"] Oct 31 01:09:01.866580 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.866546 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-token\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.866709 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.866658 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkg78\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-kube-api-access-nkg78\") pod \"e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:01.960365 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.960331 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np"] Oct 31 01:09:01.965388 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:01.965359 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7abb95-685445b999-xx7np"] Oct 31 01:09:02.127106 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:02.127079 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:02.557932 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:02.557905 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7"] Oct 31 01:09:02.559278 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:09:02.559251 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be65041_985f_484f_a025_a2e0c8d4ca6a.slice/crio-88bde9445b9f6f8c017d42505dae090470eaeb769b0006aa9e792680dd407a24 WatchSource:0}: Error finding container 88bde9445b9f6f8c017d42505dae090470eaeb769b0006aa9e792680dd407a24: Status 404 returned error can't find the container with id 88bde9445b9f6f8c017d42505dae090470eaeb769b0006aa9e792680dd407a24 Oct 31 01:09:02.561825 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:02.561793 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:09:02.561907 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:02.561861 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:09:02.561907 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:02.561896 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:09:02.692773 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:02.692729 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" event={"ID":"3be65041-985f-484f-a025-a2e0c8d4ca6a","Type":"ContainerStarted","Data":"c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a"} Oct 31 01:09:02.692963 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:02.692795 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" event={"ID":"3be65041-985f-484f-a025-a2e0c8d4ca6a","Type":"ContainerStarted","Data":"88bde9445b9f6f8c017d42505dae090470eaeb769b0006aa9e792680dd407a24"} Oct 31 01:09:02.737262 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:02.737228 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:09:02.816029 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:02.815928 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" podStartSLOduration=1.815913955 podStartE2EDuration="1.815913955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:02.813231348 +0000 UTC m=+3975.288589771" watchObservedRunningTime="2026-10-31 01:09:02.815913955 +0000 UTC m=+3975.291272377" Oct 31 01:09:03.127903 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:03.127870 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:03.134053 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:03.134026 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:03.699873 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:03.699839 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:03.700810 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:03.700788 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:09:03.735178 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:03.735146 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3487e6ad-e165-46d6-942d-f9c25c800ba5" path="/var/lib/kubelet/pods/3487e6ad-e165-46d6-942d-f9c25c800ba5/volumes" Oct 31 01:09:03.735550 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:03.735537 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59809d8f-4894-49ce-ab6c-361e00e41285" path="/var/lib/kubelet/pods/59809d8f-4894-49ce-ab6c-361e00e41285/volumes" Oct 31 01:09:10.598061 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.598003 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" containerID="cri-o://ac94b455dec9d0b4c2c9d1b0969dccacd978049b7421ff01b2e68464d4fec123" gracePeriod=45 Oct 31 01:09:10.636296 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.636258 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": dial tcp 10.133.0.98:8000: connect: connection refused" Oct 31 01:09:10.652734 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.652697 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" containerID="cri-o://17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b" gracePeriod=45 Oct 31 01:09:10.654954 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.654921 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" containerID="cri-o://68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc" gracePeriod=45 Oct 31 01:09:10.701275 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.701242 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": dial tcp 10.133.0.81:8000: connect: connection refused" Oct 31 01:09:10.765206 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.765133 4969 generic.go:356] "Generic (PLEG): container finished" podID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerID="ac94b455dec9d0b4c2c9d1b0969dccacd978049b7421ff01b2e68464d4fec123" exitCode=0 Oct 31 01:09:10.765279 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.765206 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" event={"ID":"8a680cdc-06a0-432d-9684-c642d32c7fd3","Type":"ContainerDied","Data":"ac94b455dec9d0b4c2c9d1b0969dccacd978049b7421ff01b2e68464d4fec123"} Oct 31 01:09:10.939389 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.939364 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 01:09:10.944868 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.944839 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 01:09:10.949145 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:10.949123 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 01:09:11.008978 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.008939 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-home\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-home\") pod \"8a680cdc-06a0-432d-9684-c642d32c7fd3\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " Oct 31 01:09:11.009188 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009015 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-dshm\") pod \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " Oct 31 01:09:11.009188 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009062 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-model-cache\") pod \"8a680cdc-06a0-432d-9684-c642d32c7fd3\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " Oct 31 01:09:11.009188 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009148 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-model-cache\") pod \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " Oct 31 01:09:11.009311 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009223 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tls-certs\") pod \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " Oct 31 01:09:11.009311 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009240 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-home" pod "8a680cdc-06a0-432d-9684-c642d32c7fd3" (UID: "8a680cdc-06a0-432d-9684-c642d32c7fd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.009311 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009250 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-tmp-dir\") pod \"8a680cdc-06a0-432d-9684-c642d32c7fd3\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " Oct 31 01:09:11.009522 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009335 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd96c4a-8321-47f2-88d1-1e2b281ab675-kube-api-access-22phv\" (UniqueName: \"kubernetes.io/projected/ebd96c4a-8321-47f2-88d1-1e2b281ab675-kube-api-access-22phv\") pod \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " Oct 31 01:09:11.009522 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009367 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-home\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-home\") pod \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " Oct 31 01:09:11.009522 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009399 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tmp-dir\") pod \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " Oct 31 01:09:11.009522 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009501 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-model-cache\") pod \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " Oct 31 01:09:11.009771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009544 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a680cdc-06a0-432d-9684-c642d32c7fd3-kube-api-access-ttjsm\" (UniqueName: \"kubernetes.io/projected/8a680cdc-06a0-432d-9684-c642d32c7fd3-kube-api-access-ttjsm\") pod \"8a680cdc-06a0-432d-9684-c642d32c7fd3\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " Oct 31 01:09:11.009771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009544 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-model-cache" pod "8a680cdc-06a0-432d-9684-c642d32c7fd3" (UID: "8a680cdc-06a0-432d-9684-c642d32c7fd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.009771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009548 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-model-cache" pod "ebd96c4a-8321-47f2-88d1-1e2b281ab675" (UID: "ebd96c4a-8321-47f2-88d1-1e2b281ab675"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.009771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009574 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-dshm\") pod \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " Oct 31 01:09:11.009771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009669 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-dshm\") pod \"8a680cdc-06a0-432d-9684-c642d32c7fd3\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " Oct 31 01:09:11.009771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009703 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tmp-dir\") pod \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " Oct 31 01:09:11.009771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009734 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-home\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-home\") pod \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " Oct 31 01:09:11.009771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009765 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tls-certs\") pod \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\" (UID: \"ebd96c4a-8321-47f2-88d1-1e2b281ab675\") " Oct 31 01:09:11.009771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009770 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tmp-dir" pod "525b9ee7-9d4e-48e6-b5e8-653b18b656d4" (UID: "525b9ee7-9d4e-48e6-b5e8-653b18b656d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.010428 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009784 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-model-cache" pod "525b9ee7-9d4e-48e6-b5e8-653b18b656d4" (UID: "525b9ee7-9d4e-48e6-b5e8-653b18b656d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.010428 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009805 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-kube-api-access-ck6jh\" (UniqueName: \"kubernetes.io/projected/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-kube-api-access-ck6jh\") pod \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\" (UID: \"525b9ee7-9d4e-48e6-b5e8-653b18b656d4\") " Oct 31 01:09:11.010428 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009810 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-home" pod "525b9ee7-9d4e-48e6-b5e8-653b18b656d4" (UID: "525b9ee7-9d4e-48e6-b5e8-653b18b656d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.010428 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009837 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a680cdc-06a0-432d-9684-c642d32c7fd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a680cdc-06a0-432d-9684-c642d32c7fd3-tls-certs\") pod \"8a680cdc-06a0-432d-9684-c642d32c7fd3\" (UID: \"8a680cdc-06a0-432d-9684-c642d32c7fd3\") " Oct 31 01:09:11.010428 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.009979 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-home" pod "ebd96c4a-8321-47f2-88d1-1e2b281ab675" (UID: "ebd96c4a-8321-47f2-88d1-1e2b281ab675"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.010775 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.010463 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-tmp-dir" pod "8a680cdc-06a0-432d-9684-c642d32c7fd3" (UID: "8a680cdc-06a0-432d-9684-c642d32c7fd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.010775 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.010581 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.010775 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.010632 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.010775 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.010649 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.010775 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.010664 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.010775 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.010667 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tmp-dir" pod "ebd96c4a-8321-47f2-88d1-1e2b281ab675" (UID: "ebd96c4a-8321-47f2-88d1-1e2b281ab675"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.010775 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.010680 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.010775 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.010695 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.010775 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.010707 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.012954 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.012923 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-dshm" pod "8a680cdc-06a0-432d-9684-c642d32c7fd3" (UID: "8a680cdc-06a0-432d-9684-c642d32c7fd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.013096 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.012992 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tls-certs" pod "525b9ee7-9d4e-48e6-b5e8-653b18b656d4" (UID: "525b9ee7-9d4e-48e6-b5e8-653b18b656d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:11.013096 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.013044 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a680cdc-06a0-432d-9684-c642d32c7fd3-tls-certs" pod "8a680cdc-06a0-432d-9684-c642d32c7fd3" (UID: "8a680cdc-06a0-432d-9684-c642d32c7fd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:11.013220 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.013166 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-dshm" pod "525b9ee7-9d4e-48e6-b5e8-653b18b656d4" (UID: "525b9ee7-9d4e-48e6-b5e8-653b18b656d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.013933 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.013905 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-kube-api-access-ck6jh" pod "525b9ee7-9d4e-48e6-b5e8-653b18b656d4" (UID: "525b9ee7-9d4e-48e6-b5e8-653b18b656d4"). InnerVolumeSpecName "kube-api-access-ck6jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:11.014479 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.014455 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-dshm" pod "ebd96c4a-8321-47f2-88d1-1e2b281ab675" (UID: "ebd96c4a-8321-47f2-88d1-1e2b281ab675"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:11.014619 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.014576 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd96c4a-8321-47f2-88d1-1e2b281ab675-kube-api-access-22phv" pod "ebd96c4a-8321-47f2-88d1-1e2b281ab675" (UID: "ebd96c4a-8321-47f2-88d1-1e2b281ab675"). InnerVolumeSpecName "kube-api-access-22phv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:11.014914 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.014890 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tls-certs" pod "ebd96c4a-8321-47f2-88d1-1e2b281ab675" (UID: "ebd96c4a-8321-47f2-88d1-1e2b281ab675"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:11.015810 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.015784 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a680cdc-06a0-432d-9684-c642d32c7fd3-kube-api-access-ttjsm" pod "8a680cdc-06a0-432d-9684-c642d32c7fd3" (UID: "8a680cdc-06a0-432d-9684-c642d32c7fd3"). InnerVolumeSpecName "kube-api-access-ttjsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:11.111379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111304 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111330 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111340 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111349 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22phv\" (UniqueName: \"kubernetes.io/projected/ebd96c4a-8321-47f2-88d1-1e2b281ab675-kube-api-access-22phv\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111359 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttjsm\" (UniqueName: \"kubernetes.io/projected/8a680cdc-06a0-432d-9684-c642d32c7fd3-kube-api-access-ttjsm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111367 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111375 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a680cdc-06a0-432d-9684-c642d32c7fd3-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111382 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111389 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd96c4a-8321-47f2-88d1-1e2b281ab675-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111396 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6jh\" (UniqueName: \"kubernetes.io/projected/525b9ee7-9d4e-48e6-b5e8-653b18b656d4-kube-api-access-ck6jh\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.111820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.111404 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a680cdc-06a0-432d-9684-c642d32c7fd3-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:11.392517 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.392482 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd"] Oct 31 01:09:11.393077 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393061 4969 cpu_manager.go:414] "containerMap: removing container" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" Oct 31 01:09:11.393127 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393079 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" Oct 31 01:09:11.393127 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393103 4969 cpu_manager.go:414] "containerMap: removing container" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" Oct 31 01:09:11.393127 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393108 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" Oct 31 01:09:11.393127 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393120 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" Oct 31 01:09:11.393127 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393125 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" Oct 31 01:09:11.393291 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393199 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerName="main" Oct 31 01:09:11.393291 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393210 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fbb87e-7f00-4ca7-8d11-2b7ec19cbab5" containerName="istio-proxy" Oct 31 01:09:11.393291 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393219 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerName="main" Oct 31 01:09:11.393291 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.393226 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" containerName="main" Oct 31 01:09:11.397819 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.397800 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:11.450878 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.450852 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d7412d-dockercfg-62rfv\"" Oct 31 01:09:11.455190 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.455166 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd"] Oct 31 01:09:11.489619 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.469526 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d7412d\"" Oct 31 01:09:11.515663 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.515626 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dnx\" (UniqueName: \"kubernetes.io/projected/c081aa19-9694-46ff-baec-3f22ec602f39-kube-api-access-b2dnx\") pod \"payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd\" (UID: \"c081aa19-9694-46ff-baec-3f22ec602f39\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:11.515862 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.515755 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c081aa19-9694-46ff-baec-3f22ec602f39-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd\" (UID: \"c081aa19-9694-46ff-baec-3f22ec602f39\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:11.552254 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.552222 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk"] Oct 31 01:09:11.556270 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.556251 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:11.614798 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.614763 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk"] Oct 31 01:09:11.617145 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.617062 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlx6v\" (UniqueName: \"kubernetes.io/projected/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-kube-api-access-mlx6v\") pod \"payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk\" (UID: \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:11.617145 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.617127 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk\" (UID: \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:11.617389 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.617277 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dnx\" (UniqueName: \"kubernetes.io/projected/c081aa19-9694-46ff-baec-3f22ec602f39-kube-api-access-b2dnx\") pod \"payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd\" (UID: \"c081aa19-9694-46ff-baec-3f22ec602f39\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:11.617466 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.617394 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c081aa19-9694-46ff-baec-3f22ec602f39-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd\" (UID: \"c081aa19-9694-46ff-baec-3f22ec602f39\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:11.618070 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.618052 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c081aa19-9694-46ff-baec-3f22ec602f39-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd\" (UID: \"c081aa19-9694-46ff-baec-3f22ec602f39\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:11.680446 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.680381 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dnx\" (UniqueName: \"kubernetes.io/projected/c081aa19-9694-46ff-baec-3f22ec602f39-kube-api-access-b2dnx\") pod \"payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd\" (UID: \"c081aa19-9694-46ff-baec-3f22ec602f39\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:11.711381 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.711350 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:11.718917 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.718889 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlx6v\" (UniqueName: \"kubernetes.io/projected/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-kube-api-access-mlx6v\") pod \"payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk\" (UID: \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:11.719028 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.719003 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk\" (UID: \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:11.719683 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.719661 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk\" (UID: \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:11.780931 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.779861 4969 generic.go:356] "Generic (PLEG): container finished" podID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" containerID="17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b" exitCode=0 Oct 31 01:09:11.780931 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.780037 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" event={"ID":"525b9ee7-9d4e-48e6-b5e8-653b18b656d4","Type":"ContainerDied","Data":"17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b"} Oct 31 01:09:11.780931 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.780079 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" event={"ID":"525b9ee7-9d4e-48e6-b5e8-653b18b656d4","Type":"ContainerDied","Data":"1e6212fab3bff969ed2e46f35f22d8f25915cfb467594f7c8d729c891bb1d27a"} Oct 31 01:09:11.780931 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.780103 4969 scope.go:122] "RemoveContainer" containerID="17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b" Oct 31 01:09:11.780931 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.780275 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26" Oct 31 01:09:11.788305 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.788274 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlx6v\" (UniqueName: \"kubernetes.io/projected/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-kube-api-access-mlx6v\") pod \"payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk\" (UID: \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:11.789868 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.789841 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" event={"ID":"8a680cdc-06a0-432d-9684-c642d32c7fd3","Type":"ContainerDied","Data":"b245762519a34acbd179160b6b309105babb00e7568f01c3e9393c7681da026b"} Oct 31 01:09:11.789970 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.789948 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x" Oct 31 01:09:11.792371 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.792232 4969 generic.go:356] "Generic (PLEG): container finished" podID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" containerID="68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc" exitCode=0 Oct 31 01:09:11.792371 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.792275 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" Oct 31 01:09:11.792371 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.792322 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" event={"ID":"ebd96c4a-8321-47f2-88d1-1e2b281ab675","Type":"ContainerDied","Data":"68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc"} Oct 31 01:09:11.792371 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.792354 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk" event={"ID":"ebd96c4a-8321-47f2-88d1-1e2b281ab675","Type":"ContainerDied","Data":"b0916847b0be4b9fcee9891c5c8050980f950634790a9113509d675854e6b579"} Oct 31 01:09:11.809158 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.809139 4969 scope.go:122] "RemoveContainer" containerID="17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b" Oct 31 01:09:11.809804 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:11.809459 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b\": container with ID starting with 17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b not found: ID does not exist" containerID="17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b" Oct 31 01:09:11.809804 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.809491 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b"} err="failed to get container status \"17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b\": rpc error: code = NotFound desc = could not find container \"17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b\": container with ID starting with 17a8a345eb54e953a4bea3c66b0a1d3a6b2a26a7dc07a75bdfccf70ec51c114b not found: ID does not exist" Oct 31 01:09:11.809804 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.809509 4969 scope.go:122] "RemoveContainer" containerID="ac94b455dec9d0b4c2c9d1b0969dccacd978049b7421ff01b2e68464d4fec123" Oct 31 01:09:11.818974 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.818941 4969 scope.go:122] "RemoveContainer" containerID="68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc" Oct 31 01:09:11.827781 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.827754 4969 scope.go:122] "RemoveContainer" containerID="68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc" Oct 31 01:09:11.828063 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:11.828031 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc\": container with ID starting with 68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc not found: ID does not exist" containerID="68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc" Oct 31 01:09:11.828123 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.828071 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc"} err="failed to get container status \"68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc\": rpc error: code = NotFound desc = could not find container \"68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc\": container with ID starting with 68575de2611f1430ba7c21eae509be96121ae52f3d7e45094cb27d69bb9f59bc not found: ID does not exist" Oct 31 01:09:11.871176 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.871150 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:11.947511 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.946783 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26"] Oct 31 01:09:11.949715 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:11.949692 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/premium-w2-e1ca71-kserve-549b9556d6-zmc26"] Oct 31 01:09:12.056675 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.056646 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x"] Oct 31 01:09:12.061320 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.061296 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/distinct-2-w2-e1ca71-kserve-6f958854-w2m8x"] Oct 31 01:09:12.088981 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.088959 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd"] Oct 31 01:09:12.090288 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:09:12.090260 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc081aa19_9694_46ff_baec_3f22ec602f39.slice/crio-8cab00affacc45c7ca15128310a7cb474a98fed15aa0d939dd98e7538afa1f0d WatchSource:0}: Error finding container 8cab00affacc45c7ca15128310a7cb474a98fed15aa0d939dd98e7538afa1f0d: Status 404 returned error can't find the container with id 8cab00affacc45c7ca15128310a7cb474a98fed15aa0d939dd98e7538afa1f0d Oct 31 01:09:12.170167 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.170143 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk"] Oct 31 01:09:12.174821 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.174800 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e1ca71/simulator-w2-e1ca71-kserve-74f4d589d7-n8mkk"] Oct 31 01:09:12.252624 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.252562 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk"] Oct 31 01:09:12.257723 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:09:12.257699 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cac6a39_4906_4ae7_b7ab_dbc7ccff4161.slice/crio-7a28ac17279a0b740d4e2a4d4e612a78429075fdfba45792579d4a137d894d07 WatchSource:0}: Error finding container 7a28ac17279a0b740d4e2a4d4e612a78429075fdfba45792579d4a137d894d07: Status 404 returned error can't find the container with id 7a28ac17279a0b740d4e2a4d4e612a78429075fdfba45792579d4a137d894d07 Oct 31 01:09:12.801326 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.801270 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" event={"ID":"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161","Type":"ContainerStarted","Data":"d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615"} Oct 31 01:09:12.801825 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.801334 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" event={"ID":"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161","Type":"ContainerStarted","Data":"7a28ac17279a0b740d4e2a4d4e612a78429075fdfba45792579d4a137d894d07"} Oct 31 01:09:12.801825 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.801464 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:12.803521 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.803492 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" event={"ID":"c081aa19-9694-46ff-baec-3f22ec602f39","Type":"ContainerStarted","Data":"8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886"} Oct 31 01:09:12.803691 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.803531 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" event={"ID":"c081aa19-9694-46ff-baec-3f22ec602f39","Type":"ContainerStarted","Data":"8cab00affacc45c7ca15128310a7cb474a98fed15aa0d939dd98e7538afa1f0d"} Oct 31 01:09:12.803691 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.803608 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:12.918257 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:12.918206 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" podStartSLOduration=1.918192988 podStartE2EDuration="1.918192988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:12.915869301 +0000 UTC m=+3985.391227725" watchObservedRunningTime="2026-10-31 01:09:12.918192988 +0000 UTC m=+3985.393551408" Oct 31 01:09:13.032494 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:13.032431 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" podStartSLOduration=2.032413996 podStartE2EDuration="2.032413996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:13.030560252 +0000 UTC m=+3985.505918694" watchObservedRunningTime="2026-10-31 01:09:13.032413996 +0000 UTC m=+3985.507772420" Oct 31 01:09:13.736503 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:13.736471 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525b9ee7-9d4e-48e6-b5e8-653b18b656d4" path="/var/lib/kubelet/pods/525b9ee7-9d4e-48e6-b5e8-653b18b656d4/volumes" Oct 31 01:09:13.736975 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:13.736959 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a680cdc-06a0-432d-9684-c642d32c7fd3" path="/var/lib/kubelet/pods/8a680cdc-06a0-432d-9684-c642d32c7fd3/volumes" Oct 31 01:09:13.737395 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:13.737379 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd96c4a-8321-47f2-88d1-1e2b281ab675" path="/var/lib/kubelet/pods/ebd96c4a-8321-47f2-88d1-1e2b281ab675/volumes" Oct 31 01:09:20.055673 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.055629 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx"] Oct 31 01:09:20.056258 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.055910 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" podUID="e7d0357c-9c5f-40c1-87ba-9190a0de2551" containerName="maas-api" containerID="cri-o://25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658" gracePeriod=30 Oct 31 01:09:20.317400 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.317334 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:09:20.409925 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.409894 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls\") pod \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " Oct 31 01:09:20.410090 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.409967 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d0357c-9c5f-40c1-87ba-9190a0de2551-kube-api-access-z6vhz\" (UniqueName: \"kubernetes.io/projected/e7d0357c-9c5f-40c1-87ba-9190a0de2551-kube-api-access-z6vhz\") pod \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " Oct 31 01:09:20.410090 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.410073 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-metrics-certs\") pod \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " Oct 31 01:09:20.410189 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.410172 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d0357c-9c5f-40c1-87ba-9190a0de2551-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d0357c-9c5f-40c1-87ba-9190a0de2551-openshift-service-ca\") pod \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\" (UID: \"e7d0357c-9c5f-40c1-87ba-9190a0de2551\") " Oct 31 01:09:20.410533 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.410499 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d0357c-9c5f-40c1-87ba-9190a0de2551-openshift-service-ca" pod "e7d0357c-9c5f-40c1-87ba-9190a0de2551" (UID: "e7d0357c-9c5f-40c1-87ba-9190a0de2551"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:20.410772 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.410671 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d0357c-9c5f-40c1-87ba-9190a0de2551-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.412162 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.412140 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls" pod "e7d0357c-9c5f-40c1-87ba-9190a0de2551" (UID: "e7d0357c-9c5f-40c1-87ba-9190a0de2551"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:20.412812 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.412785 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-metrics-certs" pod "e7d0357c-9c5f-40c1-87ba-9190a0de2551" (UID: "e7d0357c-9c5f-40c1-87ba-9190a0de2551"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:20.412915 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.412790 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d0357c-9c5f-40c1-87ba-9190a0de2551-kube-api-access-z6vhz" pod "e7d0357c-9c5f-40c1-87ba-9190a0de2551" (UID: "e7d0357c-9c5f-40c1-87ba-9190a0de2551"). InnerVolumeSpecName "kube-api-access-z6vhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:20.511969 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.511934 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.511969 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.511966 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6vhz\" (UniqueName: \"kubernetes.io/projected/e7d0357c-9c5f-40c1-87ba-9190a0de2551-kube-api-access-z6vhz\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.511969 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.511976 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d0357c-9c5f-40c1-87ba-9190a0de2551-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.860690 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.860655 4969 generic.go:356] "Generic (PLEG): container finished" podID="e7d0357c-9c5f-40c1-87ba-9190a0de2551" containerID="25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658" exitCode=0 Oct 31 01:09:20.860910 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.860699 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" Oct 31 01:09:20.860910 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.860747 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" event={"ID":"e7d0357c-9c5f-40c1-87ba-9190a0de2551","Type":"ContainerDied","Data":"25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658"} Oct 31 01:09:20.860910 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.860798 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx" event={"ID":"e7d0357c-9c5f-40c1-87ba-9190a0de2551","Type":"ContainerDied","Data":"6a6ed4bb3451d782f88b0f216104567c9a3b985c88766b98a04087104078af5b"} Oct 31 01:09:20.860910 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.860822 4969 scope.go:122] "RemoveContainer" containerID="25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658" Oct 31 01:09:20.871796 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.871778 4969 scope.go:122] "RemoveContainer" containerID="25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658" Oct 31 01:09:20.872094 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:20.872076 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658\": container with ID starting with 25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658 not found: ID does not exist" containerID="25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658" Oct 31 01:09:20.872165 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:20.872102 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658"} err="failed to get container status \"25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658\": rpc error: code = NotFound desc = could not find container \"25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658\": container with ID starting with 25c73ab8872fba7f285e54f42724d20e4d2067fcbbc92aa905e626f74f672658 not found: ID does not exist" Oct 31 01:09:21.029948 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:21.029907 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx"] Oct 31 01:09:21.079264 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:21.079234 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-395b06-856679976c-rkpnx"] Oct 31 01:09:21.742374 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:21.742339 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d0357c-9c5f-40c1-87ba-9190a0de2551" path="/var/lib/kubelet/pods/e7d0357c-9c5f-40c1-87ba-9190a0de2551/volumes" Oct 31 01:09:23.816550 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:23.816520 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:09:23.817006 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:23.816586 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:09:25.701508 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:25.701464 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc"] Oct 31 01:09:25.817960 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:25.817917 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85"] Oct 31 01:09:30.702489 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:30.702430 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" podUID="72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" containerName="payload-processing" containerID="cri-o://ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0" gracePeriod=25 Oct 31 01:09:30.818697 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:30.818647 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" podUID="814a5540-13fc-4f69-b185-f71fea2f7fdd" containerName="payload-pre-processing" containerID="cri-o://13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd" gracePeriod=25 Oct 31 01:09:32.727030 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:32.726997 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:09:40.425025 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:40.424989 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" podUID="814a5540-13fc-4f69-b185-f71fea2f7fdd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:09:40.425490 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:40.424990 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" podUID="72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:09:42.030941 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:42.030910 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp"] Oct 31 01:09:47.031494 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:47.031448 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" podUID="092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" containerName="payload-pre-processing" containerID="cri-o://29fd64bbb9237c5438cbc0349f34345b8cff357121983a0e077c214e8cc867c2" gracePeriod=25 Oct 31 01:09:50.425195 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:50.425152 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" podUID="814a5540-13fc-4f69-b185-f71fea2f7fdd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:09:50.425195 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:50.425187 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" podUID="72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:09:50.978902 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:50.978877 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 01:09:51.031055 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.031020 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-plugins-config-volume\") pod \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\" (UID: \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\") " Oct 31 01:09:51.031256 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.031165 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-kube-api-access-qxwqb\" (UniqueName: \"kubernetes.io/projected/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-kube-api-access-qxwqb\") pod \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\" (UID: \"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3\") " Oct 31 01:09:51.031501 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.031473 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-plugins-config-volume" pod "72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" (UID: "72bf1d7a-2f8b-4463-9e58-3cf37e1beed3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:51.031653 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.031633 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:51.034832 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.034806 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-kube-api-access-qxwqb" pod "72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" (UID: "72bf1d7a-2f8b-4463-9e58-3cf37e1beed3"). InnerVolumeSpecName "kube-api-access-qxwqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:51.065922 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.065899 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 01:09:51.073309 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.073285 4969 generic.go:356] "Generic (PLEG): container finished" podID="814a5540-13fc-4f69-b185-f71fea2f7fdd" containerID="13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd" exitCode=0 Oct 31 01:09:51.073426 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.073322 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" Oct 31 01:09:51.073426 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.073381 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" event={"ID":"814a5540-13fc-4f69-b185-f71fea2f7fdd","Type":"ContainerDied","Data":"13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd"} Oct 31 01:09:51.073558 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.073445 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85" event={"ID":"814a5540-13fc-4f69-b185-f71fea2f7fdd","Type":"ContainerDied","Data":"32538084202f70fdf653a9ae011e4d13219b04acc1c8dc4d3cecd6d78e50dd75"} Oct 31 01:09:51.073558 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.073491 4969 scope.go:122] "RemoveContainer" containerID="13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd" Oct 31 01:09:51.075046 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.075021 4969 generic.go:356] "Generic (PLEG): container finished" podID="72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" containerID="ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0" exitCode=0 Oct 31 01:09:51.075136 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.075053 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" event={"ID":"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3","Type":"ContainerDied","Data":"ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0"} Oct 31 01:09:51.075136 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.075065 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" Oct 31 01:09:51.075136 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.075079 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc" event={"ID":"72bf1d7a-2f8b-4463-9e58-3cf37e1beed3","Type":"ContainerDied","Data":"afca777a8782a08d824c0d61ac1cec58a763b78bd109d6e53b4c7f8868a62805"} Oct 31 01:09:51.083558 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.083545 4969 scope.go:122] "RemoveContainer" containerID="13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd" Oct 31 01:09:51.083846 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:51.083830 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd\": container with ID starting with 13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd not found: ID does not exist" containerID="13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd" Oct 31 01:09:51.083894 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.083853 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd"} err="failed to get container status \"13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd\": rpc error: code = NotFound desc = could not find container \"13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd\": container with ID starting with 13c697b620a14eba7251cb0097361bd6ded42088c0a14f7067b7aeaca18443bd not found: ID does not exist" Oct 31 01:09:51.083894 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.083868 4969 scope.go:122] "RemoveContainer" containerID="ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0" Oct 31 01:09:51.092867 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.092843 4969 scope.go:122] "RemoveContainer" containerID="ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0" Oct 31 01:09:51.093117 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:51.093097 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0\": container with ID starting with ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0 not found: ID does not exist" containerID="ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0" Oct 31 01:09:51.093172 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.093123 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0"} err="failed to get container status \"ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0\": rpc error: code = NotFound desc = could not find container \"ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0\": container with ID starting with ef9565deb9e412ce8ecb1bc904be99ab9baf7eb7034898d519dc15bcc290b7c0 not found: ID does not exist" Oct 31 01:09:51.132889 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.132856 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814a5540-13fc-4f69-b185-f71fea2f7fdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814a5540-13fc-4f69-b185-f71fea2f7fdd-plugins-config-volume\") pod \"814a5540-13fc-4f69-b185-f71fea2f7fdd\" (UID: \"814a5540-13fc-4f69-b185-f71fea2f7fdd\") " Oct 31 01:09:51.133071 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.133008 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814a5540-13fc-4f69-b185-f71fea2f7fdd-kube-api-access-f292z\" (UniqueName: \"kubernetes.io/projected/814a5540-13fc-4f69-b185-f71fea2f7fdd-kube-api-access-f292z\") pod \"814a5540-13fc-4f69-b185-f71fea2f7fdd\" (UID: \"814a5540-13fc-4f69-b185-f71fea2f7fdd\") " Oct 31 01:09:51.133240 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.133217 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814a5540-13fc-4f69-b185-f71fea2f7fdd-plugins-config-volume" pod "814a5540-13fc-4f69-b185-f71fea2f7fdd" (UID: "814a5540-13fc-4f69-b185-f71fea2f7fdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:51.133452 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.133426 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814a5540-13fc-4f69-b185-f71fea2f7fdd-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:51.133452 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.133453 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxwqb\" (UniqueName: \"kubernetes.io/projected/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3-kube-api-access-qxwqb\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:51.135404 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.135384 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814a5540-13fc-4f69-b185-f71fea2f7fdd-kube-api-access-f292z" pod "814a5540-13fc-4f69-b185-f71fea2f7fdd" (UID: "814a5540-13fc-4f69-b185-f71fea2f7fdd"). InnerVolumeSpecName "kube-api-access-f292z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:51.234536 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.234454 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f292z\" (UniqueName: \"kubernetes.io/projected/814a5540-13fc-4f69-b185-f71fea2f7fdd-kube-api-access-f292z\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:09:51.320984 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.320952 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc"] Oct 31 01:09:51.325572 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.325543 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-869cca-687964869c-4j2vc"] Oct 31 01:09:51.495410 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.495323 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85"] Oct 31 01:09:51.501368 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.501340 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-869cca-5f75cd56c-bhs85"] Oct 31 01:09:51.662160 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662130 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl"] Oct 31 01:09:51.662680 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662666 4969 cpu_manager.go:414] "containerMap: removing container" podUID="e7d0357c-9c5f-40c1-87ba-9190a0de2551" containerName="maas-api" Oct 31 01:09:51.662737 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662682 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d0357c-9c5f-40c1-87ba-9190a0de2551" containerName="maas-api" Oct 31 01:09:51.662737 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662715 4969 cpu_manager.go:414] "containerMap: removing container" podUID="72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" containerName="payload-processing" Oct 31 01:09:51.662737 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662720 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" containerName="payload-processing" Oct 31 01:09:51.662737 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662736 4969 cpu_manager.go:414] "containerMap: removing container" podUID="814a5540-13fc-4f69-b185-f71fea2f7fdd" containerName="payload-pre-processing" Oct 31 01:09:51.662876 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662741 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814a5540-13fc-4f69-b185-f71fea2f7fdd" containerName="payload-pre-processing" Oct 31 01:09:51.662876 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662821 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="814a5540-13fc-4f69-b185-f71fea2f7fdd" containerName="payload-pre-processing" Oct 31 01:09:51.662876 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662832 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d0357c-9c5f-40c1-87ba-9190a0de2551" containerName="maas-api" Oct 31 01:09:51.662876 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.662844 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" containerName="payload-processing" Oct 31 01:09:51.667504 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.667488 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.721279 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.721255 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d7412d-serving-cert\"" Oct 31 01:09:51.726318 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.726294 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl"] Oct 31 01:09:51.736243 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.736213 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bf1d7a-2f8b-4463-9e58-3cf37e1beed3" path="/var/lib/kubelet/pods/72bf1d7a-2f8b-4463-9e58-3cf37e1beed3/volumes" Oct 31 01:09:51.736897 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.736875 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814a5540-13fc-4f69-b185-f71fea2f7fdd" path="/var/lib/kubelet/pods/814a5540-13fc-4f69-b185-f71fea2f7fdd/volumes" Oct 31 01:09:51.739347 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.739318 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.739469 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.739367 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.739469 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.739397 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hjg\" (UniqueName: \"kubernetes.io/projected/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-kube-api-access-j4hjg\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.739572 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.739482 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-metrics-certs\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.840451 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.840358 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.840451 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.840400 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.840451 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.840421 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hjg\" (UniqueName: \"kubernetes.io/projected/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-kube-api-access-j4hjg\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.840451 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.840439 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-metrics-certs\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.840796 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:51.840523 4969 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-serving-cert: secret "maas-api-e2e-extmod-b-d7412d-serving-cert" not found Oct 31 01:09:51.840796 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:09:51.840623 4969 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls podName:eb4af30a-872b-4c82-a91a-aaa4d8a7226c nodeName:}" failed. No retries permitted until 2026-10-31 01:09:52.340582611 +0000 UTC m=+4024.815941015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls") pod "maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" (UID: "eb4af30a-872b-4c82-a91a-aaa4d8a7226c") : secret "maas-api-e2e-extmod-b-d7412d-serving-cert" not found Oct 31 01:09:51.841130 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.841103 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.843356 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.843331 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-metrics-certs\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:51.903867 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:51.903831 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hjg\" (UniqueName: \"kubernetes.io/projected/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-kube-api-access-j4hjg\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:52.345672 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:52.345628 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:52.348242 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:52.348222 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:52.580829 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:52.580793 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:52.963305 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:52.963269 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl"] Oct 31 01:09:52.964287 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:09:52.964259 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4af30a_872b_4c82_a91a_aaa4d8a7226c.slice/crio-7181ad903199570f500a8de7d3b64d800d424cbe6fbf2ccfd9b36a4eb2ec8e9e WatchSource:0}: Error finding container 7181ad903199570f500a8de7d3b64d800d424cbe6fbf2ccfd9b36a4eb2ec8e9e: Status 404 returned error can't find the container with id 7181ad903199570f500a8de7d3b64d800d424cbe6fbf2ccfd9b36a4eb2ec8e9e Oct 31 01:09:52.965746 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:52.965727 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:09:53.093468 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:53.093435 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" event={"ID":"eb4af30a-872b-4c82-a91a-aaa4d8a7226c","Type":"ContainerStarted","Data":"7181ad903199570f500a8de7d3b64d800d424cbe6fbf2ccfd9b36a4eb2ec8e9e"} Oct 31 01:09:54.102126 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:54.102095 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" event={"ID":"eb4af30a-872b-4c82-a91a-aaa4d8a7226c","Type":"ContainerStarted","Data":"7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82"} Oct 31 01:09:54.102493 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:54.102243 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:09:54.231233 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:54.231165 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" podStartSLOduration=2.197897281 podStartE2EDuration="3.231150454s" totalImagesPullingTime="1.033253173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:54.227762862 +0000 UTC m=+4026.703121284" watchObservedRunningTime="2026-10-31 01:09:54.231150454 +0000 UTC m=+4026.706508898" Oct 31 01:09:54.983754 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:09:54.983708 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" podUID="092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:10:00.115273 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:00.115240 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:10:02.741386 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:10:02.741351 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:10:05.001934 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:05.001893 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" podUID="092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:10:07.212499 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:07.212463 4969 generic.go:356] "Generic (PLEG): container finished" podID="092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" containerID="29fd64bbb9237c5438cbc0349f34345b8cff357121983a0e077c214e8cc867c2" exitCode=0 Oct 31 01:10:07.212940 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:07.212533 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" event={"ID":"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc","Type":"ContainerDied","Data":"29fd64bbb9237c5438cbc0349f34345b8cff357121983a0e077c214e8cc867c2"} Oct 31 01:10:07.339363 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:07.339338 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 01:10:07.439141 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:07.439105 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-kube-api-access-bwgkm\") pod \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\" (UID: \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\") " Oct 31 01:10:07.439323 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:07.439202 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-plugins-config-volume\") pod \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\" (UID: \"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc\") " Oct 31 01:10:07.439641 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:07.439620 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-plugins-config-volume" pod "092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" (UID: "092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:07.441409 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:07.441389 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-kube-api-access-bwgkm" pod "092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" (UID: "092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc"). InnerVolumeSpecName "kube-api-access-bwgkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:07.540148 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:07.540111 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-kube-api-access-bwgkm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:07.540148 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:07.540142 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:08.221200 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:08.221171 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" Oct 31 01:10:08.221704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:08.221176 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp" event={"ID":"092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc","Type":"ContainerDied","Data":"3f87a967d754b9a386e954220ed5f58f8e0ae6041f8deed1962f63e90e56a22f"} Oct 31 01:10:08.221704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:08.221305 4969 scope.go:122] "RemoveContainer" containerID="29fd64bbb9237c5438cbc0349f34345b8cff357121983a0e077c214e8cc867c2" Oct 31 01:10:08.402510 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:08.402473 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp"] Oct 31 01:10:08.435677 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:08.435633 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfc055-5547dd6bbc-cpkfp"] Oct 31 01:10:09.737322 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:09.737279 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" path="/var/lib/kubelet/pods/092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc/volumes" Oct 31 01:10:12.569779 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:10:12.569743 4969 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be65041_985f_484f_a025_a2e0c8d4ca6a.slice/crio-c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:10:13.259922 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:13.259891 4969 generic.go:356] "Generic (PLEG): container finished" podID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerID="c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a" exitCode=0 Oct 31 01:10:13.260118 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:13.259965 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" event={"ID":"3be65041-985f-484f-a025-a2e0c8d4ca6a","Type":"ContainerDied","Data":"c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a"} Oct 31 01:10:13.260393 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:13.260377 4969 scope.go:122] "RemoveContainer" containerID="c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a" Oct 31 01:10:13.261831 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:13.261803 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:10:13.261922 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:13.261876 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:10:13.261922 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:13.261909 4969 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:10:14.267976 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:14.267945 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" event={"ID":"3be65041-985f-484f-a025-a2e0c8d4ca6a","Type":"ContainerStarted","Data":"00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3"} Oct 31 01:10:15.127892 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:15.127857 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:10:15.129075 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:15.129049 4969 patch_prober.go:29] interesting pod/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 31 01:10:15.129180 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:15.129112 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 31 01:10:16.128393 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:16.128356 4969 patch_prober.go:29] interesting pod/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 31 01:10:16.128823 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:16.128414 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 31 01:10:17.128279 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:17.128242 4969 patch_prober.go:29] interesting pod/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 31 01:10:17.128477 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:17.128308 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 31 01:10:18.127647 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:18.127560 4969 patch_prober.go:29] interesting pod/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 31 01:10:18.127889 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:18.127666 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 31 01:10:18.701764 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:18.701717 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:10:19.128018 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:19.127981 4969 patch_prober.go:29] interesting pod/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 31 01:10:19.128216 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:19.128038 4969 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 31 01:10:20.132475 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:20.132443 4969 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:10:20.133430 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:20.133408 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:10:23.414252 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.414220 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6"] Oct 31 01:10:23.414858 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.414418 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" podUID="08677af3-2a63-4aee-8507-95a1eab3cfcf" containerName="maas-api" containerID="cri-o://f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9" gracePeriod=30 Oct 31 01:10:23.674926 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.674865 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 01:10:23.710377 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.710348 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08677af3-2a63-4aee-8507-95a1eab3cfcf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08677af3-2a63-4aee-8507-95a1eab3cfcf-openshift-service-ca\") pod \"08677af3-2a63-4aee-8507-95a1eab3cfcf\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " Oct 31 01:10:23.710511 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.710430 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls\") pod \"08677af3-2a63-4aee-8507-95a1eab3cfcf\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " Oct 31 01:10:23.710647 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.710628 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08677af3-2a63-4aee-8507-95a1eab3cfcf-kube-api-access-ntg9r\" (UniqueName: \"kubernetes.io/projected/08677af3-2a63-4aee-8507-95a1eab3cfcf-kube-api-access-ntg9r\") pod \"08677af3-2a63-4aee-8507-95a1eab3cfcf\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " Oct 31 01:10:23.710711 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.710674 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-metrics-certs\") pod \"08677af3-2a63-4aee-8507-95a1eab3cfcf\" (UID: \"08677af3-2a63-4aee-8507-95a1eab3cfcf\") " Oct 31 01:10:23.710780 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.710739 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08677af3-2a63-4aee-8507-95a1eab3cfcf-openshift-service-ca" pod "08677af3-2a63-4aee-8507-95a1eab3cfcf" (UID: "08677af3-2a63-4aee-8507-95a1eab3cfcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:23.711028 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.711007 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08677af3-2a63-4aee-8507-95a1eab3cfcf-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:23.712780 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.712756 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls" pod "08677af3-2a63-4aee-8507-95a1eab3cfcf" (UID: "08677af3-2a63-4aee-8507-95a1eab3cfcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:23.712899 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.712877 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-metrics-certs" pod "08677af3-2a63-4aee-8507-95a1eab3cfcf" (UID: "08677af3-2a63-4aee-8507-95a1eab3cfcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:23.713049 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.713033 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08677af3-2a63-4aee-8507-95a1eab3cfcf-kube-api-access-ntg9r" pod "08677af3-2a63-4aee-8507-95a1eab3cfcf" (UID: "08677af3-2a63-4aee-8507-95a1eab3cfcf"). InnerVolumeSpecName "kube-api-access-ntg9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:23.812662 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.812630 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntg9r\" (UniqueName: \"kubernetes.io/projected/08677af3-2a63-4aee-8507-95a1eab3cfcf-kube-api-access-ntg9r\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:23.812662 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.812662 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:23.812862 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:23.812677 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08677af3-2a63-4aee-8507-95a1eab3cfcf-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:24.362224 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:24.362187 4969 generic.go:356] "Generic (PLEG): container finished" podID="08677af3-2a63-4aee-8507-95a1eab3cfcf" containerID="f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9" exitCode=0 Oct 31 01:10:24.362448 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:24.362232 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" Oct 31 01:10:24.362448 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:24.362259 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" event={"ID":"08677af3-2a63-4aee-8507-95a1eab3cfcf","Type":"ContainerDied","Data":"f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9"} Oct 31 01:10:24.362448 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:24.362311 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6" event={"ID":"08677af3-2a63-4aee-8507-95a1eab3cfcf","Type":"ContainerDied","Data":"ad321a5b27e33a30306796558950d49dc454c089c97df1d4274bb0c672a51869"} Oct 31 01:10:24.362448 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:24.362343 4969 scope.go:122] "RemoveContainer" containerID="f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9" Oct 31 01:10:24.373030 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:24.373010 4969 scope.go:122] "RemoveContainer" containerID="f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9" Oct 31 01:10:24.373296 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:10:24.373277 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9\": container with ID starting with f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9 not found: ID does not exist" containerID="f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9" Oct 31 01:10:24.373372 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:24.373310 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9"} err="failed to get container status \"f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9\": rpc error: code = NotFound desc = could not find container \"f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9\": container with ID starting with f1e4b5715db7c5bd57cecd6690069686b4a6cf403316d2892bc80544197b11f9 not found: ID does not exist" Oct 31 01:10:24.529258 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:24.529218 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6"] Oct 31 01:10:24.579327 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:24.579297 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cfc055-f44979bf9-frqt6"] Oct 31 01:10:25.736317 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:25.736285 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08677af3-2a63-4aee-8507-95a1eab3cfcf" path="/var/lib/kubelet/pods/08677af3-2a63-4aee-8507-95a1eab3cfcf/volumes" Oct 31 01:10:32.609191 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:32.609156 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x"] Oct 31 01:10:32.609624 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:32.609388 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" containerID="cri-o://9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45" gracePeriod=30 Oct 31 01:10:32.729182 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:10:32.729146 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:10:35.112075 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:35.112041 4969 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:10:35.112494 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:35.112105 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:10:37.863908 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.863884 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 01:10:37.960845 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.960744 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-socket\") pod \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " Oct 31 01:10:37.960845 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.960798 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-token\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-token\") pod \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " Oct 31 01:10:37.961088 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.960855 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istiod-ca-cert\") pod \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " Oct 31 01:10:37.961088 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.960987 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-kube-api-access-n7qts\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-kube-api-access-n7qts\") pod \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " Oct 31 01:10:37.961088 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961021 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-socket" pod "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" (UID: "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:37.961272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961028 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-credential-socket\") pod \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " Oct 31 01:10:37.961272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961160 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-certs\") pod \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " Oct 31 01:10:37.961272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961197 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-envoy\") pod \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " Oct 31 01:10:37.961426 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961273 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-podinfo\") pod \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " Oct 31 01:10:37.961426 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961313 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-data\") pod \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\" (UID: \"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44\") " Oct 31 01:10:37.961426 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961326 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istiod-ca-cert" pod "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" (UID: "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:37.961426 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961344 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-credential-socket" pod "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" (UID: "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:37.961426 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961379 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-certs" pod "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" (UID: "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:37.961695 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961641 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-data" pod "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" (UID: "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:37.961780 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961761 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.961859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961787 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.961859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961803 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.961859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961819 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.961859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.961832 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.963556 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.963531 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-kube-api-access-n7qts" pod "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" (UID: "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44"). InnerVolumeSpecName "kube-api-access-n7qts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:37.963556 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.963542 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-envoy" pod "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" (UID: "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:10:37.963718 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.963529 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-podinfo" pod "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" (UID: "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:10:37.963718 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:37.963565 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-token" pod "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" (UID: "b4c3fc30-57ad-4e6f-9254-d8ee253d8d44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:38.062957 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.062915 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:38.062957 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.062951 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7qts\" (UniqueName: \"kubernetes.io/projected/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-kube-api-access-n7qts\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:38.062957 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.062966 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:38.063261 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.062974 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:38.457436 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.457401 4969 generic.go:356] "Generic (PLEG): container finished" podID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerID="9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45" exitCode=0 Oct 31 01:10:38.457642 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.457448 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" Oct 31 01:10:38.457642 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.457486 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" event={"ID":"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44","Type":"ContainerDied","Data":"9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45"} Oct 31 01:10:38.457642 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.457543 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x" event={"ID":"b4c3fc30-57ad-4e6f-9254-d8ee253d8d44","Type":"ContainerDied","Data":"d3aeb8ea600df95b5fd99636aa7176d85473edff0deff2359c69240aadeb3dad"} Oct 31 01:10:38.457642 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.457571 4969 scope.go:122] "RemoveContainer" containerID="9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45" Oct 31 01:10:38.469817 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.469794 4969 scope.go:122] "RemoveContainer" containerID="52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447" Oct 31 01:10:38.478947 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.478923 4969 scope.go:122] "RemoveContainer" containerID="9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45" Oct 31 01:10:38.479191 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:10:38.479174 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45\": container with ID starting with 9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45 not found: ID does not exist" containerID="9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45" Oct 31 01:10:38.479261 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.479200 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45"} err="failed to get container status \"9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45\": rpc error: code = NotFound desc = could not find container \"9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45\": container with ID starting with 9fdfc23dc3dd16dc34a196efc688e4d49301494543c391f06c74d1d89d77ec45 not found: ID does not exist" Oct 31 01:10:38.479261 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.479219 4969 scope.go:122] "RemoveContainer" containerID="52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447" Oct 31 01:10:38.479464 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:10:38.479447 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447\": container with ID starting with 52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447 not found: ID does not exist" containerID="52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447" Oct 31 01:10:38.479513 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.479468 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447"} err="failed to get container status \"52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447\": rpc error: code = NotFound desc = could not find container \"52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447\": container with ID starting with 52aeea083ef8a8a2a9032210a3b1588e4454bef3b229c2b70bf5d4141a764447 not found: ID does not exist" Oct 31 01:10:38.624233 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.624199 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x"] Oct 31 01:10:38.674008 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:38.673984 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfc055-openshift-default-75ff966979-7wl5x"] Oct 31 01:10:39.735286 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:10:39.735250 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" path="/var/lib/kubelet/pods/b4c3fc30-57ad-4e6f-9254-d8ee253d8d44/volumes" Oct 31 01:11:02.727159 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:11:02.727060 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:11:20.099469 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.099427 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57"] Oct 31 01:11:20.100004 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.099705 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" podUID="902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" containerName="maas-api" containerID="cri-o://7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6" gracePeriod=30 Oct 31 01:11:20.378068 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.378038 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 01:11:20.500004 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.499972 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-openshift-service-ca\") pod \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " Oct 31 01:11:20.500180 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.500107 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-kube-api-access-dq76n\" (UniqueName: \"kubernetes.io/projected/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-kube-api-access-dq76n\") pod \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " Oct 31 01:11:20.500180 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.500150 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-metrics-certs\") pod \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " Oct 31 01:11:20.500281 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.500208 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls\") pod \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\" (UID: \"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f\") " Oct 31 01:11:20.500415 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.500392 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-openshift-service-ca" pod "902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" (UID: "902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:20.500745 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.500718 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.502575 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.502545 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls" pod "902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" (UID: "902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:20.502575 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.502554 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-kube-api-access-dq76n" pod "902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" (UID: "902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f"). InnerVolumeSpecName "kube-api-access-dq76n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:20.502773 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.502730 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-metrics-certs" pod "902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" (UID: "902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:20.601640 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.601543 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq76n\" (UniqueName: \"kubernetes.io/projected/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-kube-api-access-dq76n\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.601640 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.601574 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.601640 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.601584 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.727791 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.727759 4969 generic.go:356] "Generic (PLEG): container finished" podID="902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" containerID="7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6" exitCode=0 Oct 31 01:11:20.727995 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.727804 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" Oct 31 01:11:20.727995 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.727840 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" event={"ID":"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f","Type":"ContainerDied","Data":"7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6"} Oct 31 01:11:20.727995 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.727898 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57" event={"ID":"902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f","Type":"ContainerDied","Data":"dc9793ee3b93d5986c4879ecc0eda82f53c0eebcf9b39c7c9b6c59b4d3b6a001"} Oct 31 01:11:20.727995 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.727924 4969 scope.go:122] "RemoveContainer" containerID="7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6" Oct 31 01:11:20.741156 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.741137 4969 scope.go:122] "RemoveContainer" containerID="7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6" Oct 31 01:11:20.741470 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:11:20.741448 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6\": container with ID starting with 7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6 not found: ID does not exist" containerID="7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6" Oct 31 01:11:20.741534 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.741478 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6"} err="failed to get container status \"7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6\": rpc error: code = NotFound desc = could not find container \"7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6\": container with ID starting with 7256c655e2f765a5af4e427d27a70c4533b0c6fcc886764fe0855cf34fe983f6 not found: ID does not exist" Oct 31 01:11:20.892389 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.892361 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57"] Oct 31 01:11:20.896162 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:20.896139 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c6b3b9-5b4f578897-d5h57"] Oct 31 01:11:21.734478 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:21.734446 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" path="/var/lib/kubelet/pods/902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f/volumes" Oct 31 01:11:32.728330 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:11:32.728292 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:11:35.196712 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:35.196673 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75"] Oct 31 01:11:41.676044 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:41.676005 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h"] Oct 31 01:11:41.783457 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:41.783414 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8"] Oct 31 01:11:46.676670 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:46.676624 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" podUID="1c7b692f-c20b-4754-ad19-c0fad273c3cf" containerName="payload-processing" containerID="cri-o://de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a" gracePeriod=25 Oct 31 01:11:46.784379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:46.784338 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" podUID="cd9eb718-1650-4ba0-9a5c-362cffb9354e" containerName="payload-pre-processing" containerID="cri-o://38b711041bf7efdaf04d6ff556465b62263182f5e65f8f3c82fb5c8f658a4174" gracePeriod=25 Oct 31 01:11:50.217769 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.217729 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" containerID="cri-o://25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6" gracePeriod=45 Oct 31 01:11:50.485264 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.485197 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 01:11:50.609917 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.609880 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-model-cache\") pod \"5524343d-7ab6-4683-bf9b-c2658d396a49\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " Oct 31 01:11:50.610125 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.609971 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-dshm\") pod \"5524343d-7ab6-4683-bf9b-c2658d396a49\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " Oct 31 01:11:50.610125 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.610005 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-tmp-dir\") pod \"5524343d-7ab6-4683-bf9b-c2658d396a49\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " Oct 31 01:11:50.610125 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.610050 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-home\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-home\") pod \"5524343d-7ab6-4683-bf9b-c2658d396a49\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " Oct 31 01:11:50.610338 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.610194 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-model-cache" pod "5524343d-7ab6-4683-bf9b-c2658d396a49" (UID: "5524343d-7ab6-4683-bf9b-c2658d396a49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.610398 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.610325 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-tmp-dir" pod "5524343d-7ab6-4683-bf9b-c2658d396a49" (UID: "5524343d-7ab6-4683-bf9b-c2658d396a49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.610460 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.610395 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-home" pod "5524343d-7ab6-4683-bf9b-c2658d396a49" (UID: "5524343d-7ab6-4683-bf9b-c2658d396a49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.610520 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.610468 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5524343d-7ab6-4683-bf9b-c2658d396a49-tls-certs\" (UniqueName: \"kubernetes.io/secret/5524343d-7ab6-4683-bf9b-c2658d396a49-tls-certs\") pod \"5524343d-7ab6-4683-bf9b-c2658d396a49\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " Oct 31 01:11:50.610624 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.610576 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5524343d-7ab6-4683-bf9b-c2658d396a49-kube-api-access-d2g8b\" (UniqueName: \"kubernetes.io/projected/5524343d-7ab6-4683-bf9b-c2658d396a49-kube-api-access-d2g8b\") pod \"5524343d-7ab6-4683-bf9b-c2658d396a49\" (UID: \"5524343d-7ab6-4683-bf9b-c2658d396a49\") " Oct 31 01:11:50.611039 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.610980 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.611039 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.611007 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.611039 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.611019 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.613188 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.613145 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5524343d-7ab6-4683-bf9b-c2658d396a49-tls-certs" pod "5524343d-7ab6-4683-bf9b-c2658d396a49" (UID: "5524343d-7ab6-4683-bf9b-c2658d396a49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:50.613480 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.613453 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5524343d-7ab6-4683-bf9b-c2658d396a49-kube-api-access-d2g8b" pod "5524343d-7ab6-4683-bf9b-c2658d396a49" (UID: "5524343d-7ab6-4683-bf9b-c2658d396a49"). InnerVolumeSpecName "kube-api-access-d2g8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:50.613678 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.613657 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-dshm" pod "5524343d-7ab6-4683-bf9b-c2658d396a49" (UID: "5524343d-7ab6-4683-bf9b-c2658d396a49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:50.712436 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.712405 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2g8b\" (UniqueName: \"kubernetes.io/projected/5524343d-7ab6-4683-bf9b-c2658d396a49-kube-api-access-d2g8b\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.712436 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.712431 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5524343d-7ab6-4683-bf9b-c2658d396a49-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.712681 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.712444 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5524343d-7ab6-4683-bf9b-c2658d396a49-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:50.917360 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.917326 4969 generic.go:356] "Generic (PLEG): container finished" podID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerID="25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6" exitCode=0 Oct 31 01:11:50.917360 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.917372 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" Oct 31 01:11:50.917654 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.917408 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" event={"ID":"5524343d-7ab6-4683-bf9b-c2658d396a49","Type":"ContainerDied","Data":"25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6"} Oct 31 01:11:50.917654 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.917453 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75" event={"ID":"5524343d-7ab6-4683-bf9b-c2658d396a49","Type":"ContainerDied","Data":"e28dd46bac64b19fc6ece433c15548d0f91f4ee6fe2367420f9ec448b81ce749"} Oct 31 01:11:50.917654 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.917485 4969 scope.go:122] "RemoveContainer" containerID="25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6" Oct 31 01:11:50.928517 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.928500 4969 scope.go:122] "RemoveContainer" containerID="25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6" Oct 31 01:11:50.928909 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:11:50.928890 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6\": container with ID starting with 25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6 not found: ID does not exist" containerID="25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6" Oct 31 01:11:50.928963 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:50.928917 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6"} err="failed to get container status \"25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6\": rpc error: code = NotFound desc = could not find container \"25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6\": container with ID starting with 25c65d69c5d7a7f247eed836501fb6dbca9db74ad39e193764919baf4baa0ee6 not found: ID does not exist" Oct 31 01:11:51.081654 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:51.081624 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75"] Oct 31 01:11:51.131281 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:51.131245 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/simulator-w0-10a7aa-kserve-9c4f87cb6-vvj75"] Oct 31 01:11:51.734468 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:51.734432 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" path="/var/lib/kubelet/pods/5524343d-7ab6-4683-bf9b-c2658d396a49/volumes" Oct 31 01:11:54.135812 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:54.135769 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" podUID="cd9eb718-1650-4ba0-9a5c-362cffb9354e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:11:54.136205 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:11:54.135769 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" podUID="1c7b692f-c20b-4754-ad19-c0fad273c3cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:12:02.728299 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:12:02.728256 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:12:04.136240 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:04.136202 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" podUID="1c7b692f-c20b-4754-ad19-c0fad273c3cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:12:04.136787 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:04.136204 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" podUID="cd9eb718-1650-4ba0-9a5c-362cffb9354e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:12:06.948046 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:06.947968 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 01:12:06.985244 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:06.985211 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7b692f-c20b-4754-ad19-c0fad273c3cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7b692f-c20b-4754-ad19-c0fad273c3cf-plugins-config-volume\") pod \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\" (UID: \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\") " Oct 31 01:12:06.985380 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:06.985276 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7b692f-c20b-4754-ad19-c0fad273c3cf-kube-api-access-25pbh\" (UniqueName: \"kubernetes.io/projected/1c7b692f-c20b-4754-ad19-c0fad273c3cf-kube-api-access-25pbh\") pod \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\" (UID: \"1c7b692f-c20b-4754-ad19-c0fad273c3cf\") " Oct 31 01:12:06.985847 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:06.985813 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7b692f-c20b-4754-ad19-c0fad273c3cf-plugins-config-volume" pod "1c7b692f-c20b-4754-ad19-c0fad273c3cf" (UID: "1c7b692f-c20b-4754-ad19-c0fad273c3cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:06.988459 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:06.988433 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7b692f-c20b-4754-ad19-c0fad273c3cf-kube-api-access-25pbh" pod "1c7b692f-c20b-4754-ad19-c0fad273c3cf" (UID: "1c7b692f-c20b-4754-ad19-c0fad273c3cf"). InnerVolumeSpecName "kube-api-access-25pbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:07.024667 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.024638 4969 generic.go:356] "Generic (PLEG): container finished" podID="cd9eb718-1650-4ba0-9a5c-362cffb9354e" containerID="38b711041bf7efdaf04d6ff556465b62263182f5e65f8f3c82fb5c8f658a4174" exitCode=0 Oct 31 01:12:07.024803 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.024721 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" event={"ID":"cd9eb718-1650-4ba0-9a5c-362cffb9354e","Type":"ContainerDied","Data":"38b711041bf7efdaf04d6ff556465b62263182f5e65f8f3c82fb5c8f658a4174"} Oct 31 01:12:07.026306 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.026277 4969 generic.go:356] "Generic (PLEG): container finished" podID="1c7b692f-c20b-4754-ad19-c0fad273c3cf" containerID="de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a" exitCode=0 Oct 31 01:12:07.026425 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.026319 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" Oct 31 01:12:07.026425 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.026343 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" event={"ID":"1c7b692f-c20b-4754-ad19-c0fad273c3cf","Type":"ContainerDied","Data":"de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a"} Oct 31 01:12:07.026425 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.026388 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h" event={"ID":"1c7b692f-c20b-4754-ad19-c0fad273c3cf","Type":"ContainerDied","Data":"94760dd9a7a0ab5fb93fe6f27a34bedd0446ea06ddf043a75c554af55c58e33c"} Oct 31 01:12:07.026425 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.026416 4969 scope.go:122] "RemoveContainer" containerID="de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a" Oct 31 01:12:07.032301 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.032283 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 01:12:07.037861 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.037845 4969 scope.go:122] "RemoveContainer" containerID="de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a" Oct 31 01:12:07.038129 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:12:07.038112 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a\": container with ID starting with de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a not found: ID does not exist" containerID="de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a" Oct 31 01:12:07.038171 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.038137 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a"} err="failed to get container status \"de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a\": rpc error: code = NotFound desc = could not find container \"de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a\": container with ID starting with de42398c5113d6c39e0e7644edf469f6d6efaaf22624c2a289321e73c2866b5a not found: ID does not exist" Oct 31 01:12:07.086540 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.086504 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9eb718-1650-4ba0-9a5c-362cffb9354e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb718-1650-4ba0-9a5c-362cffb9354e-plugins-config-volume\") pod \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\" (UID: \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\") " Oct 31 01:12:07.086783 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.086763 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9eb718-1650-4ba0-9a5c-362cffb9354e-kube-api-access-ll7gq\" (UniqueName: \"kubernetes.io/projected/cd9eb718-1650-4ba0-9a5c-362cffb9354e-kube-api-access-ll7gq\") pod \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\" (UID: \"cd9eb718-1650-4ba0-9a5c-362cffb9354e\") " Oct 31 01:12:07.086876 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.086855 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9eb718-1650-4ba0-9a5c-362cffb9354e-plugins-config-volume" pod "cd9eb718-1650-4ba0-9a5c-362cffb9354e" (UID: "cd9eb718-1650-4ba0-9a5c-362cffb9354e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:12:07.087180 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.087162 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25pbh\" (UniqueName: \"kubernetes.io/projected/1c7b692f-c20b-4754-ad19-c0fad273c3cf-kube-api-access-25pbh\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:07.087225 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.087184 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c7b692f-c20b-4754-ad19-c0fad273c3cf-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:07.087225 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.087195 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb718-1650-4ba0-9a5c-362cffb9354e-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:07.089173 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.089149 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9eb718-1650-4ba0-9a5c-362cffb9354e-kube-api-access-ll7gq" pod "cd9eb718-1650-4ba0-9a5c-362cffb9354e" (UID: "cd9eb718-1650-4ba0-9a5c-362cffb9354e"). InnerVolumeSpecName "kube-api-access-ll7gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:12:07.176288 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.176248 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h"] Oct 31 01:12:07.180424 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.180398 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10a7aa-5476fbbfd5-qvj9h"] Oct 31 01:12:07.188061 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.188040 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7gq\" (UniqueName: \"kubernetes.io/projected/cd9eb718-1650-4ba0-9a5c-362cffb9354e-kube-api-access-ll7gq\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:12:07.734137 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:07.734102 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7b692f-c20b-4754-ad19-c0fad273c3cf" path="/var/lib/kubelet/pods/1c7b692f-c20b-4754-ad19-c0fad273c3cf/volumes" Oct 31 01:12:08.035538 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:08.035460 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" event={"ID":"cd9eb718-1650-4ba0-9a5c-362cffb9354e","Type":"ContainerDied","Data":"82c02f16594ab49d9a40959cf54307e72891ddb3918e6f208fcd1a13bc8f791c"} Oct 31 01:12:08.035538 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:08.035495 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8" Oct 31 01:12:08.035538 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:08.035517 4969 scope.go:122] "RemoveContainer" containerID="38b711041bf7efdaf04d6ff556465b62263182f5e65f8f3c82fb5c8f658a4174" Oct 31 01:12:08.203050 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:08.203019 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8"] Oct 31 01:12:08.252375 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:08.252342 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10a7aa-59d6ccbc47-l87q8"] Oct 31 01:12:09.733909 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:09.733876 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9eb718-1650-4ba0-9a5c-362cffb9354e" path="/var/lib/kubelet/pods/cd9eb718-1650-4ba0-9a5c-362cffb9354e/volumes" Oct 31 01:12:32.726868 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:12:32.726827 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:12:48.486792 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:48.486764 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 01:12:48.503724 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:48.503697 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 01:12:48.524331 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:48.524306 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d775kfz_fc66fc26-a5d3-4165-8937-b2dae564dab9/extract/0.log" Oct 31 01:12:48.539833 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:12:48.539809 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-wqpvv_a00d1d31-58a5-4835-ad53-56c65f5adbb0/console-operator/3.log" Oct 31 01:13:02.727669 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:13:02.727633 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:13:32.727279 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:13:32.727235 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:13:38.143571 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.143539 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q"] Oct 31 01:13:38.144131 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.143765 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" podUID="0dc2efd1-88d0-4194-a9ac-9e649b242cf7" containerName="maas-api" containerID="cri-o://a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1" gracePeriod=30 Oct 31 01:13:38.410884 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.410816 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:13:38.512936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.512907 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-metrics-certs\") pod \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " Oct 31 01:13:38.513102 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.513016 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-kube-api-access-rqkm2\" (UniqueName: \"kubernetes.io/projected/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-kube-api-access-rqkm2\") pod \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " Oct 31 01:13:38.513102 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.513089 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls\") pod \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " Oct 31 01:13:38.513200 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.513125 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-openshift-service-ca\") pod \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\" (UID: \"0dc2efd1-88d0-4194-a9ac-9e649b242cf7\") " Oct 31 01:13:38.513579 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.513553 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-openshift-service-ca" pod "0dc2efd1-88d0-4194-a9ac-9e649b242cf7" (UID: "0dc2efd1-88d0-4194-a9ac-9e649b242cf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:38.515886 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.515828 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-metrics-certs" pod "0dc2efd1-88d0-4194-a9ac-9e649b242cf7" (UID: "0dc2efd1-88d0-4194-a9ac-9e649b242cf7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:38.518633 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.517426 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls" pod "0dc2efd1-88d0-4194-a9ac-9e649b242cf7" (UID: "0dc2efd1-88d0-4194-a9ac-9e649b242cf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:38.518633 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.517580 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-kube-api-access-rqkm2" pod "0dc2efd1-88d0-4194-a9ac-9e649b242cf7" (UID: "0dc2efd1-88d0-4194-a9ac-9e649b242cf7"). InnerVolumeSpecName "kube-api-access-rqkm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:38.602439 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.602401 4969 generic.go:356] "Generic (PLEG): container finished" podID="0dc2efd1-88d0-4194-a9ac-9e649b242cf7" containerID="a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1" exitCode=0 Oct 31 01:13:38.602668 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.602444 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" Oct 31 01:13:38.602668 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.602448 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" event={"ID":"0dc2efd1-88d0-4194-a9ac-9e649b242cf7","Type":"ContainerDied","Data":"a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1"} Oct 31 01:13:38.602668 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.602576 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q" event={"ID":"0dc2efd1-88d0-4194-a9ac-9e649b242cf7","Type":"ContainerDied","Data":"e64b3f44e9802ee34816ef1af70532269c9bb445733b768ec5245ef47d5f0f02"} Oct 31 01:13:38.602668 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.602629 4969 scope.go:122] "RemoveContainer" containerID="a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1" Oct 31 01:13:38.613101 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.613082 4969 scope.go:122] "RemoveContainer" containerID="a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1" Oct 31 01:13:38.613358 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:13:38.613339 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1\": container with ID starting with a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1 not found: ID does not exist" containerID="a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1" Oct 31 01:13:38.613427 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.613366 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1"} err="failed to get container status \"a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1\": rpc error: code = NotFound desc = could not find container \"a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1\": container with ID starting with a79fefe0592a1fdd9749dde00db0c7bc9b10dc2da47610931e7bf5a4135bc2a1 not found: ID does not exist" Oct 31 01:13:38.613883 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.613866 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqkm2\" (UniqueName: \"kubernetes.io/projected/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-kube-api-access-rqkm2\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:38.613932 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.613889 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:38.613932 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.613900 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:38.613932 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.613909 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dc2efd1-88d0-4194-a9ac-9e649b242cf7-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:38.766914 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.766836 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q"] Oct 31 01:13:38.773431 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:38.773398 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3853b6-64d4c69cc4-w7p7q"] Oct 31 01:13:39.733612 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:39.733557 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc2efd1-88d0-4194-a9ac-9e649b242cf7" path="/var/lib/kubelet/pods/0dc2efd1-88d0-4194-a9ac-9e649b242cf7/volumes" Oct 31 01:13:59.334049 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:59.333970 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk"] Oct 31 01:13:59.438946 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:13:59.438918 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd"] Oct 31 01:14:02.727130 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:02.727095 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:14:04.334890 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:04.334846 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" podUID="7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" containerName="payload-processing" containerID="cri-o://d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615" gracePeriod=25 Oct 31 01:14:04.439254 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:04.439211 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" podUID="c081aa19-9694-46ff-baec-3f22ec602f39" containerName="payload-pre-processing" containerID="cri-o://8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886" gracePeriod=25 Oct 31 01:14:08.690739 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:08.690707 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl"] Oct 31 01:14:08.691168 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:08.690895 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" podUID="eb4af30a-872b-4c82-a91a-aaa4d8a7226c" containerName="maas-api" containerID="cri-o://7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82" gracePeriod=30 Oct 31 01:14:08.949166 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:08.949105 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:14:09.000924 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.000885 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-metrics-certs\") pod \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " Oct 31 01:14:09.001055 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.000942 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-kube-api-access-j4hjg\" (UniqueName: \"kubernetes.io/projected/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-kube-api-access-j4hjg\") pod \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " Oct 31 01:14:09.001055 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.000966 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-openshift-service-ca\") pod \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " Oct 31 01:14:09.001055 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.001052 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls\") pod \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\" (UID: \"eb4af30a-872b-4c82-a91a-aaa4d8a7226c\") " Oct 31 01:14:09.001812 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.001769 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-openshift-service-ca" pod "eb4af30a-872b-4c82-a91a-aaa4d8a7226c" (UID: "eb4af30a-872b-4c82-a91a-aaa4d8a7226c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:09.005805 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.004331 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls" pod "eb4af30a-872b-4c82-a91a-aaa4d8a7226c" (UID: "eb4af30a-872b-4c82-a91a-aaa4d8a7226c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:09.012659 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.012613 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-metrics-certs" pod "eb4af30a-872b-4c82-a91a-aaa4d8a7226c" (UID: "eb4af30a-872b-4c82-a91a-aaa4d8a7226c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:09.012798 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.012746 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-kube-api-access-j4hjg" pod "eb4af30a-872b-4c82-a91a-aaa4d8a7226c" (UID: "eb4af30a-872b-4c82-a91a-aaa4d8a7226c"). InnerVolumeSpecName "kube-api-access-j4hjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:09.103069 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.103033 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:09.103069 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.103061 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:09.103272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.103091 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4hjg\" (UniqueName: \"kubernetes.io/projected/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-kube-api-access-j4hjg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:09.103272 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.103101 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4af30a-872b-4c82-a91a-aaa4d8a7226c-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:09.795480 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.795450 4969 generic.go:356] "Generic (PLEG): container finished" podID="eb4af30a-872b-4c82-a91a-aaa4d8a7226c" containerID="7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82" exitCode=0 Oct 31 01:14:09.795946 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.795502 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" event={"ID":"eb4af30a-872b-4c82-a91a-aaa4d8a7226c","Type":"ContainerDied","Data":"7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82"} Oct 31 01:14:09.795946 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.795515 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" Oct 31 01:14:09.795946 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.795542 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl" event={"ID":"eb4af30a-872b-4c82-a91a-aaa4d8a7226c","Type":"ContainerDied","Data":"7181ad903199570f500a8de7d3b64d800d424cbe6fbf2ccfd9b36a4eb2ec8e9e"} Oct 31 01:14:09.795946 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.795564 4969 scope.go:122] "RemoveContainer" containerID="7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82" Oct 31 01:14:09.805549 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.805530 4969 scope.go:122] "RemoveContainer" containerID="7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82" Oct 31 01:14:09.805833 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:09.805810 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82\": container with ID starting with 7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82 not found: ID does not exist" containerID="7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82" Oct 31 01:14:09.805927 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.805840 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82"} err="failed to get container status \"7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82\": rpc error: code = NotFound desc = could not find container \"7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82\": container with ID starting with 7d21ffa6f85fdbb011dfbd34f4e7c42106b1b23832f45f4c4e11c423b8c15c82 not found: ID does not exist" Oct 31 01:14:09.962270 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:09.962225 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl"] Oct 31 01:14:10.011711 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:10.011623 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d7412d-7865fcfc6b-8rhrl"] Oct 31 01:14:11.734429 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:11.734387 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4af30a-872b-4c82-a91a-aaa4d8a7226c" path="/var/lib/kubelet/pods/eb4af30a-872b-4c82-a91a-aaa4d8a7226c/volumes" Oct 31 01:14:13.814826 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:13.814777 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" podUID="c081aa19-9694-46ff-baec-3f22ec602f39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:13.815261 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:13.814829 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" podUID="7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:17.956017 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:17.955986 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7"] Oct 31 01:14:17.956541 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:17.956187 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" containerID="cri-o://00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3" gracePeriod=30 Oct 31 01:14:20.135232 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:20.135190 4969 patch_prober.go:29] interesting pod/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:14:20.135657 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:20.135276 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:14:23.222452 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.222379 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:14:23.340679 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.340648 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-envoy\") pod \"3be65041-985f-484f-a025-a2e0c8d4ca6a\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " Oct 31 01:14:23.340849 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.340689 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-certs\") pod \"3be65041-985f-484f-a025-a2e0c8d4ca6a\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " Oct 31 01:14:23.340849 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.340738 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-podinfo\") pod \"3be65041-985f-484f-a025-a2e0c8d4ca6a\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " Oct 31 01:14:23.340849 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.340761 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-credential-socket\") pod \"3be65041-985f-484f-a025-a2e0c8d4ca6a\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " Oct 31 01:14:23.341014 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.340850 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-kube-api-access-nkg78\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-kube-api-access-nkg78\") pod \"3be65041-985f-484f-a025-a2e0c8d4ca6a\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " Oct 31 01:14:23.341014 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.340897 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-data\") pod \"3be65041-985f-484f-a025-a2e0c8d4ca6a\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " Oct 31 01:14:23.341014 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.340934 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be65041-985f-484f-a025-a2e0c8d4ca6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be65041-985f-484f-a025-a2e0c8d4ca6a-istiod-ca-cert\") pod \"3be65041-985f-484f-a025-a2e0c8d4ca6a\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " Oct 31 01:14:23.341183 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341012 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-token\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-token\") pod \"3be65041-985f-484f-a025-a2e0c8d4ca6a\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " Oct 31 01:14:23.341183 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341029 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-certs" pod "3be65041-985f-484f-a025-a2e0c8d4ca6a" (UID: "3be65041-985f-484f-a025-a2e0c8d4ca6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:23.341183 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341055 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-socket\") pod \"3be65041-985f-484f-a025-a2e0c8d4ca6a\" (UID: \"3be65041-985f-484f-a025-a2e0c8d4ca6a\") " Oct 31 01:14:23.341357 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341172 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-credential-socket" pod "3be65041-985f-484f-a025-a2e0c8d4ca6a" (UID: "3be65041-985f-484f-a025-a2e0c8d4ca6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:23.341430 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341389 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-socket" pod "3be65041-985f-484f-a025-a2e0c8d4ca6a" (UID: "3be65041-985f-484f-a025-a2e0c8d4ca6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:23.341430 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341412 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be65041-985f-484f-a025-a2e0c8d4ca6a-istiod-ca-cert" pod "3be65041-985f-484f-a025-a2e0c8d4ca6a" (UID: "3be65041-985f-484f-a025-a2e0c8d4ca6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:23.341565 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341536 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-data" pod "3be65041-985f-484f-a025-a2e0c8d4ca6a" (UID: "3be65041-985f-484f-a025-a2e0c8d4ca6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:23.341678 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341560 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be65041-985f-484f-a025-a2e0c8d4ca6a-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:23.341678 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341582 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:23.341678 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341615 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:23.341678 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.341630 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:23.343100 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.343078 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-envoy" pod "3be65041-985f-484f-a025-a2e0c8d4ca6a" (UID: "3be65041-985f-484f-a025-a2e0c8d4ca6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:23.343647 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.343626 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-token" pod "3be65041-985f-484f-a025-a2e0c8d4ca6a" (UID: "3be65041-985f-484f-a025-a2e0c8d4ca6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:23.343831 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.343806 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-kube-api-access-nkg78" pod "3be65041-985f-484f-a025-a2e0c8d4ca6a" (UID: "3be65041-985f-484f-a025-a2e0c8d4ca6a"). InnerVolumeSpecName "kube-api-access-nkg78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:23.343831 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.343814 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-podinfo" pod "3be65041-985f-484f-a025-a2e0c8d4ca6a" (UID: "3be65041-985f-484f-a025-a2e0c8d4ca6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:23.443081 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.443048 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:23.443081 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.443080 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:23.443286 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.443095 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkg78\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-kube-api-access-nkg78\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:23.443286 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.443110 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:23.443286 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.443122 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be65041-985f-484f-a025-a2e0c8d4ca6a-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:23.814861 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.814820 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" podUID="c081aa19-9694-46ff-baec-3f22ec602f39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:23.814861 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.814845 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" podUID="7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:23.885168 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.885134 4969 generic.go:356] "Generic (PLEG): container finished" podID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerID="00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3" exitCode=0 Oct 31 01:14:23.885345 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.885183 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" Oct 31 01:14:23.885345 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.885192 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" event={"ID":"3be65041-985f-484f-a025-a2e0c8d4ca6a","Type":"ContainerDied","Data":"00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3"} Oct 31 01:14:23.885345 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.885241 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7" event={"ID":"3be65041-985f-484f-a025-a2e0c8d4ca6a","Type":"ContainerDied","Data":"88bde9445b9f6f8c017d42505dae090470eaeb769b0006aa9e792680dd407a24"} Oct 31 01:14:23.885345 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.885268 4969 scope.go:122] "RemoveContainer" containerID="00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3" Oct 31 01:14:23.895174 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.895153 4969 scope.go:122] "RemoveContainer" containerID="c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a" Oct 31 01:14:23.906572 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.906549 4969 scope.go:122] "RemoveContainer" containerID="00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3" Oct 31 01:14:23.906902 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:23.906876 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3\": container with ID starting with 00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3 not found: ID does not exist" containerID="00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3" Oct 31 01:14:23.906986 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.906910 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3"} err="failed to get container status \"00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3\": rpc error: code = NotFound desc = could not find container \"00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3\": container with ID starting with 00d18b9e33e80a0941c87244a6ad40976023cb6cced7bb5da2087c0b205d71d3 not found: ID does not exist" Oct 31 01:14:23.906986 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.906934 4969 scope.go:122] "RemoveContainer" containerID="c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a" Oct 31 01:14:23.907219 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:23.907203 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a\": container with ID starting with c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a not found: ID does not exist" containerID="c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a" Oct 31 01:14:23.907270 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:23.907226 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a"} err="failed to get container status \"c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a\": rpc error: code = NotFound desc = could not find container \"c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a\": container with ID starting with c773646a7c60f42a95591d6852af4544bf3ca3ed01cf8ecb3ec12509e5059d1a not found: ID does not exist" Oct 31 01:14:24.050201 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.050121 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7"] Oct 31 01:14:24.099708 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.099677 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d7412d-openshift-default-58cf4475bf-22ql7"] Oct 31 01:14:24.595912 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.595855 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:14:24.656419 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.656390 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-kube-api-access-mlx6v\" (UniqueName: \"kubernetes.io/projected/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-kube-api-access-mlx6v\") pod \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\" (UID: \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\") " Oct 31 01:14:24.656624 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.656461 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-plugins-config-volume\") pod \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\" (UID: \"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161\") " Oct 31 01:14:24.656877 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.656853 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-plugins-config-volume" pod "7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" (UID: "7cac6a39-4906-4ae7-b7ab-dbc7ccff4161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:24.659951 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.659920 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-kube-api-access-mlx6v" pod "7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" (UID: "7cac6a39-4906-4ae7-b7ab-dbc7ccff4161"). InnerVolumeSpecName "kube-api-access-mlx6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:24.687307 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.687286 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:14:24.758055 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.758027 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c081aa19-9694-46ff-baec-3f22ec602f39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c081aa19-9694-46ff-baec-3f22ec602f39-plugins-config-volume\") pod \"c081aa19-9694-46ff-baec-3f22ec602f39\" (UID: \"c081aa19-9694-46ff-baec-3f22ec602f39\") " Oct 31 01:14:24.758211 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.758071 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c081aa19-9694-46ff-baec-3f22ec602f39-kube-api-access-b2dnx\" (UniqueName: \"kubernetes.io/projected/c081aa19-9694-46ff-baec-3f22ec602f39-kube-api-access-b2dnx\") pod \"c081aa19-9694-46ff-baec-3f22ec602f39\" (UID: \"c081aa19-9694-46ff-baec-3f22ec602f39\") " Oct 31 01:14:24.758256 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.758245 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.758293 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.758256 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlx6v\" (UniqueName: \"kubernetes.io/projected/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161-kube-api-access-mlx6v\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.758428 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.758407 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c081aa19-9694-46ff-baec-3f22ec602f39-plugins-config-volume" pod "c081aa19-9694-46ff-baec-3f22ec602f39" (UID: "c081aa19-9694-46ff-baec-3f22ec602f39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:24.760280 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.760261 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c081aa19-9694-46ff-baec-3f22ec602f39-kube-api-access-b2dnx" pod "c081aa19-9694-46ff-baec-3f22ec602f39" (UID: "c081aa19-9694-46ff-baec-3f22ec602f39"). InnerVolumeSpecName "kube-api-access-b2dnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:24.860499 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.860349 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c081aa19-9694-46ff-baec-3f22ec602f39-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.860499 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.860381 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2dnx\" (UniqueName: \"kubernetes.io/projected/c081aa19-9694-46ff-baec-3f22ec602f39-kube-api-access-b2dnx\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.893938 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.893905 4969 generic.go:356] "Generic (PLEG): container finished" podID="7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" containerID="d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615" exitCode=0 Oct 31 01:14:24.894105 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.893944 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" Oct 31 01:14:24.894105 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.893987 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" event={"ID":"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161","Type":"ContainerDied","Data":"d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615"} Oct 31 01:14:24.894105 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.894044 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk" event={"ID":"7cac6a39-4906-4ae7-b7ab-dbc7ccff4161","Type":"ContainerDied","Data":"7a28ac17279a0b740d4e2a4d4e612a78429075fdfba45792579d4a137d894d07"} Oct 31 01:14:24.894105 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.894076 4969 scope.go:122] "RemoveContainer" containerID="d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615" Oct 31 01:14:24.895391 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.895372 4969 generic.go:356] "Generic (PLEG): container finished" podID="c081aa19-9694-46ff-baec-3f22ec602f39" containerID="8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886" exitCode=0 Oct 31 01:14:24.895501 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.895420 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" Oct 31 01:14:24.895501 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.895444 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" event={"ID":"c081aa19-9694-46ff-baec-3f22ec602f39","Type":"ContainerDied","Data":"8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886"} Oct 31 01:14:24.895501 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.895495 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd" event={"ID":"c081aa19-9694-46ff-baec-3f22ec602f39","Type":"ContainerDied","Data":"8cab00affacc45c7ca15128310a7cb474a98fed15aa0d939dd98e7538afa1f0d"} Oct 31 01:14:24.915508 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.915487 4969 scope.go:122] "RemoveContainer" containerID="d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615" Oct 31 01:14:24.915844 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:24.915823 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615\": container with ID starting with d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615 not found: ID does not exist" containerID="d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615" Oct 31 01:14:24.915909 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.915857 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615"} err="failed to get container status \"d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615\": rpc error: code = NotFound desc = could not find container \"d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615\": container with ID starting with d5813dfc3ca0d7b484fdf7851f60fbd71d72262eb09ae8bd561a77b3c288f615 not found: ID does not exist" Oct 31 01:14:24.915909 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.915878 4969 scope.go:122] "RemoveContainer" containerID="8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886" Oct 31 01:14:24.925083 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.925062 4969 scope.go:122] "RemoveContainer" containerID="8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886" Oct 31 01:14:24.925338 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:24.925320 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886\": container with ID starting with 8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886 not found: ID does not exist" containerID="8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886" Oct 31 01:14:24.925385 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:24.925347 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886"} err="failed to get container status \"8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886\": rpc error: code = NotFound desc = could not find container \"8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886\": container with ID starting with 8a89b1844e708c957f9c53d4a1fccdc40ef6baae45094686b05cada79dafd886 not found: ID does not exist" Oct 31 01:14:25.059555 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:25.059520 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk"] Oct 31 01:14:25.064220 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:25.064198 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d7412d-6d7786cfb9-n6hpk"] Oct 31 01:14:25.172861 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:25.172828 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd"] Oct 31 01:14:25.176777 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:25.176749 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d7412d-6fdcd6565f-mgcrd"] Oct 31 01:14:25.733295 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:25.733262 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" path="/var/lib/kubelet/pods/3be65041-985f-484f-a025-a2e0c8d4ca6a/volumes" Oct 31 01:14:25.733700 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:25.733692 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" path="/var/lib/kubelet/pods/7cac6a39-4906-4ae7-b7ab-dbc7ccff4161/volumes" Oct 31 01:14:25.734041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:25.734028 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c081aa19-9694-46ff-baec-3f22ec602f39" path="/var/lib/kubelet/pods/c081aa19-9694-46ff-baec-3f22ec602f39/volumes" Oct 31 01:14:32.727074 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:32.727035 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:14:44.234161 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.234128 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs"] Oct 31 01:14:44.234647 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.234355 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" podUID="7807fb24-677e-40cd-baa5-e0d023dd47dc" containerName="maas-api" containerID="cri-o://9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279" gracePeriod=30 Oct 31 01:14:44.493692 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.493628 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 01:14:44.655410 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.655375 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls\") pod \"7807fb24-677e-40cd-baa5-e0d023dd47dc\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " Oct 31 01:14:44.655581 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.655443 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-metrics-certs\") pod \"7807fb24-677e-40cd-baa5-e0d023dd47dc\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " Oct 31 01:14:44.655581 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.655472 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7807fb24-677e-40cd-baa5-e0d023dd47dc-kube-api-access-d4z6w\" (UniqueName: \"kubernetes.io/projected/7807fb24-677e-40cd-baa5-e0d023dd47dc-kube-api-access-d4z6w\") pod \"7807fb24-677e-40cd-baa5-e0d023dd47dc\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " Oct 31 01:14:44.655581 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.655528 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7807fb24-677e-40cd-baa5-e0d023dd47dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7807fb24-677e-40cd-baa5-e0d023dd47dc-openshift-service-ca\") pod \"7807fb24-677e-40cd-baa5-e0d023dd47dc\" (UID: \"7807fb24-677e-40cd-baa5-e0d023dd47dc\") " Oct 31 01:14:44.655947 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.655912 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7807fb24-677e-40cd-baa5-e0d023dd47dc-openshift-service-ca" pod "7807fb24-677e-40cd-baa5-e0d023dd47dc" (UID: "7807fb24-677e-40cd-baa5-e0d023dd47dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:44.657850 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.657814 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls" pod "7807fb24-677e-40cd-baa5-e0d023dd47dc" (UID: "7807fb24-677e-40cd-baa5-e0d023dd47dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:44.657850 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.657835 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7807fb24-677e-40cd-baa5-e0d023dd47dc-kube-api-access-d4z6w" pod "7807fb24-677e-40cd-baa5-e0d023dd47dc" (UID: "7807fb24-677e-40cd-baa5-e0d023dd47dc"). InnerVolumeSpecName "kube-api-access-d4z6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:44.658011 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.657893 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-metrics-certs" pod "7807fb24-677e-40cd-baa5-e0d023dd47dc" (UID: "7807fb24-677e-40cd-baa5-e0d023dd47dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:44.756935 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.756845 4969 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7807fb24-677e-40cd-baa5-e0d023dd47dc-openshift-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:44.756935 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.756874 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-maas-api-tls\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:44.756935 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.756883 4969 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7807fb24-677e-40cd-baa5-e0d023dd47dc-metrics-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:44.756935 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:44.756901 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4z6w\" (UniqueName: \"kubernetes.io/projected/7807fb24-677e-40cd-baa5-e0d023dd47dc-kube-api-access-d4z6w\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:45.030879 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.030793 4969 generic.go:356] "Generic (PLEG): container finished" podID="7807fb24-677e-40cd-baa5-e0d023dd47dc" containerID="9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279" exitCode=0 Oct 31 01:14:45.030879 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.030844 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" Oct 31 01:14:45.031093 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.030882 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" event={"ID":"7807fb24-677e-40cd-baa5-e0d023dd47dc","Type":"ContainerDied","Data":"9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279"} Oct 31 01:14:45.031093 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.030941 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs" event={"ID":"7807fb24-677e-40cd-baa5-e0d023dd47dc","Type":"ContainerDied","Data":"c2ea7d7f1b0098755c0869ed4207aab70bec0f2d6be43eceb86bc205c36780bf"} Oct 31 01:14:45.031093 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.030969 4969 scope.go:122] "RemoveContainer" containerID="9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279" Oct 31 01:14:45.041324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.041301 4969 scope.go:122] "RemoveContainer" containerID="9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279" Oct 31 01:14:45.041620 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:45.041572 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279\": container with ID starting with 9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279 not found: ID does not exist" containerID="9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279" Oct 31 01:14:45.041702 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.041620 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279"} err="failed to get container status \"9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279\": rpc error: code = NotFound desc = could not find container \"9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279\": container with ID starting with 9bc423b752b2685d169cf70c1a0ed0beaa59d36230d94ee584c0f490c69e3279 not found: ID does not exist" Oct 31 01:14:45.195311 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.195274 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs"] Oct 31 01:14:45.245994 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.245958 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4743f8-84c7bffbd5-tddzs"] Oct 31 01:14:45.734135 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:45.734102 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7807fb24-677e-40cd-baa5-e0d023dd47dc" path="/var/lib/kubelet/pods/7807fb24-677e-40cd-baa5-e0d023dd47dc/volumes" Oct 31 01:14:53.128216 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:53.128180 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn"] Oct 31 01:14:53.128797 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:53.128384 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" containerID="cri-o://938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc" gracePeriod=30 Oct 31 01:14:54.057175 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.057136 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl"] Oct 31 01:14:54.058326 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.058297 4969 cpu_manager.go:414] "containerMap: removing container" podUID="0dc2efd1-88d0-4194-a9ac-9e649b242cf7" containerName="maas-api" Oct 31 01:14:54.058498 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.058485 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc2efd1-88d0-4194-a9ac-9e649b242cf7" containerName="maas-api" Oct 31 01:14:54.058608 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.058578 4969 cpu_manager.go:414] "containerMap: removing container" podUID="092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" containerName="payload-pre-processing" Oct 31 01:14:54.058700 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.058687 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" containerName="payload-pre-processing" Oct 31 01:14:54.058833 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.058824 4969 cpu_manager.go:414] "containerMap: removing container" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" Oct 31 01:14:54.058911 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.058900 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" Oct 31 01:14:54.058997 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.058988 4969 cpu_manager.go:414] "containerMap: removing container" podUID="7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" containerName="payload-processing" Oct 31 01:14:54.059071 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059061 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" containerName="payload-processing" Oct 31 01:14:54.059162 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059153 4969 cpu_manager.go:414] "containerMap: removing container" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" Oct 31 01:14:54.059234 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059225 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" Oct 31 01:14:54.059308 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059299 4969 cpu_manager.go:414] "containerMap: removing container" podUID="eb4af30a-872b-4c82-a91a-aaa4d8a7226c" containerName="maas-api" Oct 31 01:14:54.059375 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059366 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4af30a-872b-4c82-a91a-aaa4d8a7226c" containerName="maas-api" Oct 31 01:14:54.059459 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059451 4969 cpu_manager.go:414] "containerMap: removing container" podUID="1c7b692f-c20b-4754-ad19-c0fad273c3cf" containerName="payload-processing" Oct 31 01:14:54.059534 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059525 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7b692f-c20b-4754-ad19-c0fad273c3cf" containerName="payload-processing" Oct 31 01:14:54.059637 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059628 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" Oct 31 01:14:54.059713 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059703 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" Oct 31 01:14:54.059815 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059805 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" Oct 31 01:14:54.059880 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059871 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059958 4969 cpu_manager.go:414] "containerMap: removing container" podUID="08677af3-2a63-4aee-8507-95a1eab3cfcf" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059968 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08677af3-2a63-4aee-8507-95a1eab3cfcf" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059988 4969 cpu_manager.go:414] "containerMap: removing container" podUID="cd9eb718-1650-4ba0-9a5c-362cffb9354e" containerName="payload-pre-processing" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.059995 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9eb718-1650-4ba0-9a5c-362cffb9354e" containerName="payload-pre-processing" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060012 4969 cpu_manager.go:414] "containerMap: removing container" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060019 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060028 4969 cpu_manager.go:414] "containerMap: removing container" podUID="7807fb24-677e-40cd-baa5-e0d023dd47dc" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060040 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7807fb24-677e-40cd-baa5-e0d023dd47dc" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060051 4969 cpu_manager.go:414] "containerMap: removing container" podUID="c081aa19-9694-46ff-baec-3f22ec602f39" containerName="payload-pre-processing" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060059 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c081aa19-9694-46ff-baec-3f22ec602f39" containerName="payload-pre-processing" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060078 4969 cpu_manager.go:414] "containerMap: removing container" podUID="902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060084 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060288 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060307 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="08677af3-2a63-4aee-8507-95a1eab3cfcf" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060319 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9eb718-1650-4ba0-9a5c-362cffb9354e" containerName="payload-pre-processing" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060335 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="5524343d-7ab6-4683-bf9b-c2658d396a49" containerName="main" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060350 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="7807fb24-677e-40cd-baa5-e0d023dd47dc" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060366 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="c081aa19-9694-46ff-baec-3f22ec602f39" containerName="payload-pre-processing" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060383 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="902e7d5e-0dc8-40ac-b8c8-43ae0105ba3f" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060393 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc2efd1-88d0-4194-a9ac-9e649b242cf7" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060409 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="092b2bde-bf8c-4cf0-b41d-1bd6c323e6fc" containerName="payload-pre-processing" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060429 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060444 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cac6a39-4906-4ae7-b7ab-dbc7ccff4161" containerName="payload-processing" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060459 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be65041-985f-484f-a025-a2e0c8d4ca6a" containerName="istio-proxy" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060467 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4af30a-872b-4c82-a91a-aaa4d8a7226c" containerName="maas-api" Oct 31 01:14:54.061423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.060476 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7b692f-c20b-4754-ad19-c0fad273c3cf" containerName="payload-processing" Oct 31 01:14:54.066233 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.066213 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.117276 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.117244 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl"] Oct 31 01:14:54.137757 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.137732 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.138128 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.137880 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkm7v\" (UniqueName: \"kubernetes.io/projected/7e7d2be3-f929-46db-8a57-4e0fd43350fe-kube-api-access-tkm7v\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.138128 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.137923 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.163460 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.163437 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:54.163460 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.163437 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:14:54.238375 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.238349 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkm7v\" (UniqueName: \"kubernetes.io/projected/7e7d2be3-f929-46db-8a57-4e0fd43350fe-kube-api-access-tkm7v\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.238532 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.238390 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.238532 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.238436 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.239225 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.239163 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.241009 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.240986 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.299239 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.299212 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkm7v\" (UniqueName: \"kubernetes.io/projected/7e7d2be3-f929-46db-8a57-4e0fd43350fe-kube-api-access-tkm7v\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.378981 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.378958 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:54.753137 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:54.753108 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl"] Oct 31 01:14:54.755033 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:14:54.754998 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7d2be3_f929_46db_8a57_4e0fd43350fe.slice/crio-0c7393bc3dbb085101ab042c632dbd5ec8bfc491ad1efae63418c93537534fdb WatchSource:0}: Error finding container 0c7393bc3dbb085101ab042c632dbd5ec8bfc491ad1efae63418c93537534fdb: Status 404 returned error can't find the container with id 0c7393bc3dbb085101ab042c632dbd5ec8bfc491ad1efae63418c93537534fdb Oct 31 01:14:55.094384 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:55.094352 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" event={"ID":"7e7d2be3-f929-46db-8a57-4e0fd43350fe","Type":"ContainerStarted","Data":"0c7393bc3dbb085101ab042c632dbd5ec8bfc491ad1efae63418c93537534fdb"} Oct 31 01:14:56.102690 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:56.102585 4969 generic.go:356] "Generic (PLEG): container finished" podID="7e7d2be3-f929-46db-8a57-4e0fd43350fe" containerID="5cb8037c4dc6a80ad4412bbff930844a3dcac715a4bca6eb26539f962084a0e4" exitCode=0 Oct 31 01:14:56.102690 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:56.102622 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" event={"ID":"7e7d2be3-f929-46db-8a57-4e0fd43350fe","Type":"ContainerDied","Data":"5cb8037c4dc6a80ad4412bbff930844a3dcac715a4bca6eb26539f962084a0e4"} Oct 31 01:14:57.250193 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.250165 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:57.268352 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.268325 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7d2be3-f929-46db-8a57-4e0fd43350fe-kube-api-access-tkm7v\" (UniqueName: \"kubernetes.io/projected/7e7d2be3-f929-46db-8a57-4e0fd43350fe-kube-api-access-tkm7v\") pod \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " Oct 31 01:14:57.268468 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.268384 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-openshift-ca\") pod \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " Oct 31 01:14:57.268513 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.268468 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-xks-ca\") pod \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\" (UID: \"7e7d2be3-f929-46db-8a57-4e0fd43350fe\") " Oct 31 01:14:57.268749 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.268724 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-openshift-ca" pod "7e7d2be3-f929-46db-8a57-4e0fd43350fe" (UID: "7e7d2be3-f929-46db-8a57-4e0fd43350fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:57.268862 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.268798 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:57.270607 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.270568 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7d2be3-f929-46db-8a57-4e0fd43350fe-kube-api-access-tkm7v" pod "7e7d2be3-f929-46db-8a57-4e0fd43350fe" (UID: "7e7d2be3-f929-46db-8a57-4e0fd43350fe"). InnerVolumeSpecName "kube-api-access-tkm7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:57.270691 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.270661 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-xks-ca" pod "7e7d2be3-f929-46db-8a57-4e0fd43350fe" (UID: "7e7d2be3-f929-46db-8a57-4e0fd43350fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:57.370278 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.370244 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkm7v\" (UniqueName: \"kubernetes.io/projected/7e7d2be3-f929-46db-8a57-4e0fd43350fe-kube-api-access-tkm7v\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:57.370278 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:57.370277 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7d2be3-f929-46db-8a57-4e0fd43350fe-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:58.115374 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.115347 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" Oct 31 01:14:58.115536 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.115354 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl" event={"ID":"7e7d2be3-f929-46db-8a57-4e0fd43350fe","Type":"ContainerDied","Data":"0c7393bc3dbb085101ab042c632dbd5ec8bfc491ad1efae63418c93537534fdb"} Oct 31 01:14:58.115536 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.115462 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c7393bc3dbb085101ab042c632dbd5ec8bfc491ad1efae63418c93537534fdb" Oct 31 01:14:58.397287 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.397262 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 01:14:58.479313 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479287 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-data\") pod \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " Oct 31 01:14:58.479486 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479336 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istiod-ca-cert\") pod \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " Oct 31 01:14:58.479486 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479366 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-kube-api-access-mr24n\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-kube-api-access-mr24n\") pod \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " Oct 31 01:14:58.479486 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479387 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-envoy\") pod \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " Oct 31 01:14:58.479486 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479419 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-token\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-token\") pod \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " Oct 31 01:14:58.479486 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479441 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-certs\") pod \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " Oct 31 01:14:58.479898 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479494 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-credential-socket\") pod \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " Oct 31 01:14:58.479898 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479539 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-socket\") pod \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " Oct 31 01:14:58.479898 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479619 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-podinfo\") pod \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\" (UID: \"0dcd577f-e015-4cbc-a379-d8ba5fe22385\") " Oct 31 01:14:58.479898 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479582 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-data" pod "0dcd577f-e015-4cbc-a379-d8ba5fe22385" (UID: "0dcd577f-e015-4cbc-a379-d8ba5fe22385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:58.479898 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479861 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istiod-ca-cert" pod "0dcd577f-e015-4cbc-a379-d8ba5fe22385" (UID: "0dcd577f-e015-4cbc-a379-d8ba5fe22385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:58.480163 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479916 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-credential-socket" pod "0dcd577f-e015-4cbc-a379-d8ba5fe22385" (UID: "0dcd577f-e015-4cbc-a379-d8ba5fe22385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:58.480163 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.479938 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-certs" pod "0dcd577f-e015-4cbc-a379-d8ba5fe22385" (UID: "0dcd577f-e015-4cbc-a379-d8ba5fe22385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:58.480163 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.480083 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:58.480163 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.480103 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:58.480163 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.480104 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-socket" pod "0dcd577f-e015-4cbc-a379-d8ba5fe22385" (UID: "0dcd577f-e015-4cbc-a379-d8ba5fe22385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:58.480163 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.480118 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:58.480163 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.480134 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:58.481946 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.481916 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-token" pod "0dcd577f-e015-4cbc-a379-d8ba5fe22385" (UID: "0dcd577f-e015-4cbc-a379-d8ba5fe22385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:58.482066 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.482026 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-podinfo" pod "0dcd577f-e015-4cbc-a379-d8ba5fe22385" (UID: "0dcd577f-e015-4cbc-a379-d8ba5fe22385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:58.482111 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.482057 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-envoy" pod "0dcd577f-e015-4cbc-a379-d8ba5fe22385" (UID: "0dcd577f-e015-4cbc-a379-d8ba5fe22385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:58.482111 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.482074 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-kube-api-access-mr24n" pod "0dcd577f-e015-4cbc-a379-d8ba5fe22385" (UID: "0dcd577f-e015-4cbc-a379-d8ba5fe22385"). InnerVolumeSpecName "kube-api-access-mr24n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:58.581213 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.581180 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:58.581213 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.581215 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:58.581424 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.581229 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:58.581424 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.581241 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr24n\" (UniqueName: \"kubernetes.io/projected/0dcd577f-e015-4cbc-a379-d8ba5fe22385-kube-api-access-mr24n\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:58.581424 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:58.581254 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcd577f-e015-4cbc-a379-d8ba5fe22385-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:59.129318 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.129281 4969 generic.go:356] "Generic (PLEG): container finished" podID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerID="938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc" exitCode=0 Oct 31 01:14:59.129524 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.129353 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" Oct 31 01:14:59.129524 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.129371 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" event={"ID":"0dcd577f-e015-4cbc-a379-d8ba5fe22385","Type":"ContainerDied","Data":"938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc"} Oct 31 01:14:59.129524 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.129451 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn" event={"ID":"0dcd577f-e015-4cbc-a379-d8ba5fe22385","Type":"ContainerDied","Data":"1f5fb8d8fca6fed68f719257dbebb33507e5fbc3d497c1cbc71f4e4b41c9c0f7"} Oct 31 01:14:59.129524 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.129474 4969 scope.go:122] "RemoveContainer" containerID="938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc" Oct 31 01:14:59.140487 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.140466 4969 scope.go:122] "RemoveContainer" containerID="a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe" Oct 31 01:14:59.149285 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.149268 4969 scope.go:122] "RemoveContainer" containerID="938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc" Oct 31 01:14:59.149559 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:59.149541 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc\": container with ID starting with 938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc not found: ID does not exist" containerID="938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc" Oct 31 01:14:59.149637 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.149568 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc"} err="failed to get container status \"938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc\": rpc error: code = NotFound desc = could not find container \"938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc\": container with ID starting with 938b10094cb29dc3c530ecd37fe7594744b7166f3430be8356711297d22c0bfc not found: ID does not exist" Oct 31 01:14:59.149704 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.149586 4969 scope.go:122] "RemoveContainer" containerID="a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe" Oct 31 01:14:59.149955 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:14:59.149937 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe\": container with ID starting with a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe not found: ID does not exist" containerID="a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe" Oct 31 01:14:59.150024 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.149964 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe"} err="failed to get container status \"a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe\": rpc error: code = NotFound desc = could not find container \"a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe\": container with ID starting with a48dc6cff3fa38d28951bac5b5e591e2365b7bb79365c7378843a97614b83ebe not found: ID does not exist" Oct 31 01:14:59.343936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.343894 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn"] Oct 31 01:14:59.393777 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.393707 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4743f8-openshift-default-57c47bdf5-ng4tn"] Oct 31 01:14:59.733944 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:14:59.733868 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" path="/var/lib/kubelet/pods/0dcd577f-e015-4cbc-a379-d8ba5fe22385/volumes" Oct 31 01:15:00.161476 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.161445 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj"] Oct 31 01:15:00.161989 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.161969 4969 cpu_manager.go:414] "containerMap: removing container" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" Oct 31 01:15:00.162103 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.161991 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" Oct 31 01:15:00.162103 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.162010 4969 cpu_manager.go:414] "containerMap: removing container" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" Oct 31 01:15:00.162103 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.162017 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" Oct 31 01:15:00.162103 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.162038 4969 cpu_manager.go:414] "containerMap: removing container" podUID="7e7d2be3-f929-46db-8a57-4e0fd43350fe" containerName="revoke-keys" Oct 31 01:15:00.162103 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.162046 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7d2be3-f929-46db-8a57-4e0fd43350fe" containerName="revoke-keys" Oct 31 01:15:00.162372 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.162170 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" Oct 31 01:15:00.162372 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.162186 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcd577f-e015-4cbc-a379-d8ba5fe22385" containerName="istio-proxy" Oct 31 01:15:00.162372 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.162198 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7d2be3-f929-46db-8a57-4e0fd43350fe" containerName="revoke-keys" Oct 31 01:15:00.162372 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.162208 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c3fc30-57ad-4e6f-9254-d8ee253d8d44" containerName="istio-proxy" Oct 31 01:15:00.166809 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.166786 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn"] Oct 31 01:15:00.166935 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.166895 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:00.170207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.170188 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:00.194633 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.194612 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26wt\" (UniqueName: \"kubernetes.io/projected/b76440b9-7df1-4a68-b594-574635feba46-kube-api-access-s26wt\") pod \"maas-api-key-cleanup-29890155-bqnpj\" (UID: \"b76440b9-7df1-4a68-b594-574635feba46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:00.194756 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.194647 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a880593-ae39-4bec-810a-ee2263ea9066-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn\" (UID: \"2a880593-ae39-4bec-810a-ee2263ea9066\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:00.194756 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.194671 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76440b9-7df1-4a68-b594-574635feba46-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-bqnpj\" (UID: \"b76440b9-7df1-4a68-b594-574635feba46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:00.194889 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.194838 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knvz\" (UniqueName: \"kubernetes.io/projected/2a880593-ae39-4bec-810a-ee2263ea9066-kube-api-access-2knvz\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn\" (UID: \"2a880593-ae39-4bec-810a-ee2263ea9066\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:00.219153 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.219120 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:15:00.272687 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.272648 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj"] Oct 31 01:15:00.274364 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.274333 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn"] Oct 31 01:15:00.295417 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.295389 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26wt\" (UniqueName: \"kubernetes.io/projected/b76440b9-7df1-4a68-b594-574635feba46-kube-api-access-s26wt\") pod \"maas-api-key-cleanup-29890155-bqnpj\" (UID: \"b76440b9-7df1-4a68-b594-574635feba46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:00.295539 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.295423 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a880593-ae39-4bec-810a-ee2263ea9066-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn\" (UID: \"2a880593-ae39-4bec-810a-ee2263ea9066\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:00.295539 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.295449 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76440b9-7df1-4a68-b594-574635feba46-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-bqnpj\" (UID: \"b76440b9-7df1-4a68-b594-574635feba46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:00.295539 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.295517 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knvz\" (UniqueName: \"kubernetes.io/projected/2a880593-ae39-4bec-810a-ee2263ea9066-kube-api-access-2knvz\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn\" (UID: \"2a880593-ae39-4bec-810a-ee2263ea9066\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:00.296161 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.296141 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a880593-ae39-4bec-810a-ee2263ea9066-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn\" (UID: \"2a880593-ae39-4bec-810a-ee2263ea9066\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:00.296202 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.296140 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76440b9-7df1-4a68-b594-574635feba46-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-bqnpj\" (UID: \"b76440b9-7df1-4a68-b594-574635feba46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:00.356719 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.356694 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26wt\" (UniqueName: \"kubernetes.io/projected/b76440b9-7df1-4a68-b594-574635feba46-kube-api-access-s26wt\") pod \"maas-api-key-cleanup-29890155-bqnpj\" (UID: \"b76440b9-7df1-4a68-b594-574635feba46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:00.356719 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.356705 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knvz\" (UniqueName: \"kubernetes.io/projected/2a880593-ae39-4bec-810a-ee2263ea9066-kube-api-access-2knvz\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn\" (UID: \"2a880593-ae39-4bec-810a-ee2263ea9066\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:00.481164 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.481096 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:00.486181 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.486157 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:00.862289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.862259 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj"] Oct 31 01:15:00.862887 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:15:00.862863 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76440b9_7df1_4a68_b594_574635feba46.slice/crio-e20e06ac6339edae109e1a35a28d1833179a73a9b5e9ec434e5902aed99886ea WatchSource:0}: Error finding container e20e06ac6339edae109e1a35a28d1833179a73a9b5e9ec434e5902aed99886ea: Status 404 returned error can't find the container with id e20e06ac6339edae109e1a35a28d1833179a73a9b5e9ec434e5902aed99886ea Oct 31 01:15:00.910096 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:00.910073 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn"] Oct 31 01:15:00.910989 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:15:00.910956 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a880593_ae39_4bec_810a_ee2263ea9066.slice/crio-a7007466d60773bdef3783c9dbd211853f39c1488cbbf7657f84a940cf5e54f5 WatchSource:0}: Error finding container a7007466d60773bdef3783c9dbd211853f39c1488cbbf7657f84a940cf5e54f5: Status 404 returned error can't find the container with id a7007466d60773bdef3783c9dbd211853f39c1488cbbf7657f84a940cf5e54f5 Oct 31 01:15:01.144545 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:01.144513 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" event={"ID":"b76440b9-7df1-4a68-b594-574635feba46","Type":"ContainerStarted","Data":"e20e06ac6339edae109e1a35a28d1833179a73a9b5e9ec434e5902aed99886ea"} Oct 31 01:15:01.145697 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:01.145672 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" event={"ID":"2a880593-ae39-4bec-810a-ee2263ea9066","Type":"ContainerStarted","Data":"a7007466d60773bdef3783c9dbd211853f39c1488cbbf7657f84a940cf5e54f5"} Oct 31 01:15:02.153510 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:02.153478 4969 generic.go:356] "Generic (PLEG): container finished" podID="b76440b9-7df1-4a68-b594-574635feba46" containerID="94ea9c1abef6a51fa87ab0f122e3702667fbd3742939590125741a7c5b80f3b3" exitCode=60 Oct 31 01:15:02.153960 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:02.153523 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" event={"ID":"b76440b9-7df1-4a68-b594-574635feba46","Type":"ContainerDied","Data":"94ea9c1abef6a51fa87ab0f122e3702667fbd3742939590125741a7c5b80f3b3"} Oct 31 01:15:02.153960 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:02.153861 4969 scope.go:122] "RemoveContainer" containerID="94ea9c1abef6a51fa87ab0f122e3702667fbd3742939590125741a7c5b80f3b3" Oct 31 01:15:02.155399 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:02.155314 4969 generic.go:356] "Generic (PLEG): container finished" podID="2a880593-ae39-4bec-810a-ee2263ea9066" containerID="304969fa2053cf973abfdf8cd610bb9f3a9f507ed598537faed8d86253d18186" exitCode=60 Oct 31 01:15:02.155399 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:02.155348 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" event={"ID":"2a880593-ae39-4bec-810a-ee2263ea9066","Type":"ContainerDied","Data":"304969fa2053cf973abfdf8cd610bb9f3a9f507ed598537faed8d86253d18186"} Oct 31 01:15:02.155573 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:02.155557 4969 scope.go:122] "RemoveContainer" containerID="304969fa2053cf973abfdf8cd610bb9f3a9f507ed598537faed8d86253d18186" Oct 31 01:15:02.727000 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:15:02.726951 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:15:03.172211 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:03.172170 4969 generic.go:356] "Generic (PLEG): container finished" podID="2a880593-ae39-4bec-810a-ee2263ea9066" containerID="87372e4cc0df1b84b14d557ba018e2db677e1f3dfd7fddf8059bab169eb3814b" exitCode=60 Oct 31 01:15:03.172706 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:03.172257 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" event={"ID":"2a880593-ae39-4bec-810a-ee2263ea9066","Type":"ContainerDied","Data":"87372e4cc0df1b84b14d557ba018e2db677e1f3dfd7fddf8059bab169eb3814b"} Oct 31 01:15:03.172706 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:03.172346 4969 scope.go:122] "RemoveContainer" containerID="304969fa2053cf973abfdf8cd610bb9f3a9f507ed598537faed8d86253d18186" Oct 31 01:15:03.172706 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:03.172609 4969 scope.go:122] "RemoveContainer" containerID="87372e4cc0df1b84b14d557ba018e2db677e1f3dfd7fddf8059bab169eb3814b" Oct 31 01:15:03.172901 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:15:03.172873 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn_odh-ai-gateway-infra(2a880593-ae39-4bec-810a-ee2263ea9066)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" podUID="2a880593-ae39-4bec-810a-ee2263ea9066" Oct 31 01:15:03.174695 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:03.174672 4969 generic.go:356] "Generic (PLEG): container finished" podID="b76440b9-7df1-4a68-b594-574635feba46" containerID="6e3ddcd1b09643145e15f62e38a85c7bc88885d9c255ecb6112278381dcdf32a" exitCode=60 Oct 31 01:15:03.174795 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:03.174738 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" event={"ID":"b76440b9-7df1-4a68-b594-574635feba46","Type":"ContainerDied","Data":"6e3ddcd1b09643145e15f62e38a85c7bc88885d9c255ecb6112278381dcdf32a"} Oct 31 01:15:03.174948 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:03.174937 4969 scope.go:122] "RemoveContainer" containerID="6e3ddcd1b09643145e15f62e38a85c7bc88885d9c255ecb6112278381dcdf32a" Oct 31 01:15:03.175140 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:15:03.175126 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-bqnpj_odh-ai-gateway-infra(b76440b9-7df1-4a68-b594-574635feba46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" podUID="b76440b9-7df1-4a68-b594-574635feba46" Oct 31 01:15:03.188384 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:03.188363 4969 scope.go:122] "RemoveContainer" containerID="94ea9c1abef6a51fa87ab0f122e3702667fbd3742939590125741a7c5b80f3b3" Oct 31 01:15:04.183484 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:04.183453 4969 scope.go:122] "RemoveContainer" containerID="87372e4cc0df1b84b14d557ba018e2db677e1f3dfd7fddf8059bab169eb3814b" Oct 31 01:15:04.183941 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:15:04.183747 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn_odh-ai-gateway-infra(2a880593-ae39-4bec-810a-ee2263ea9066)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" podUID="2a880593-ae39-4bec-810a-ee2263ea9066" Oct 31 01:15:04.185103 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:04.185084 4969 scope.go:122] "RemoveContainer" containerID="6e3ddcd1b09643145e15f62e38a85c7bc88885d9c255ecb6112278381dcdf32a" Oct 31 01:15:04.185294 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:15:04.185277 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-bqnpj_odh-ai-gateway-infra(b76440b9-7df1-4a68-b594-574635feba46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" podUID="b76440b9-7df1-4a68-b594-574635feba46" Oct 31 01:15:14.844642 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:14.844608 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn"] Oct 31 01:15:14.978854 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:14.978826 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:15.034305 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.034273 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a880593-ae39-4bec-810a-ee2263ea9066-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a880593-ae39-4bec-810a-ee2263ea9066-maas-api-service-ca\") pod \"2a880593-ae39-4bec-810a-ee2263ea9066\" (UID: \"2a880593-ae39-4bec-810a-ee2263ea9066\") " Oct 31 01:15:15.034465 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.034326 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a880593-ae39-4bec-810a-ee2263ea9066-kube-api-access-2knvz\" (UniqueName: \"kubernetes.io/projected/2a880593-ae39-4bec-810a-ee2263ea9066-kube-api-access-2knvz\") pod \"2a880593-ae39-4bec-810a-ee2263ea9066\" (UID: \"2a880593-ae39-4bec-810a-ee2263ea9066\") " Oct 31 01:15:15.034658 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.034632 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a880593-ae39-4bec-810a-ee2263ea9066-maas-api-service-ca" pod "2a880593-ae39-4bec-810a-ee2263ea9066" (UID: "2a880593-ae39-4bec-810a-ee2263ea9066"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:15.036633 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.036587 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a880593-ae39-4bec-810a-ee2263ea9066-kube-api-access-2knvz" pod "2a880593-ae39-4bec-810a-ee2263ea9066" (UID: "2a880593-ae39-4bec-810a-ee2263ea9066"). InnerVolumeSpecName "kube-api-access-2knvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:15.135548 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.135515 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a880593-ae39-4bec-810a-ee2263ea9066-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:15.135548 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.135546 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knvz\" (UniqueName: \"kubernetes.io/projected/2a880593-ae39-4bec-810a-ee2263ea9066-kube-api-access-2knvz\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:15.256881 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.256850 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" Oct 31 01:15:15.257071 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.256853 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn" event={"ID":"2a880593-ae39-4bec-810a-ee2263ea9066","Type":"ContainerDied","Data":"a7007466d60773bdef3783c9dbd211853f39c1488cbbf7657f84a940cf5e54f5"} Oct 31 01:15:15.257071 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.256976 4969 scope.go:122] "RemoveContainer" containerID="87372e4cc0df1b84b14d557ba018e2db677e1f3dfd7fddf8059bab169eb3814b" Oct 31 01:15:15.420944 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.420861 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn"] Oct 31 01:15:15.427143 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.427120 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-093220-29890155-9zgsn"] Oct 31 01:15:15.727771 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.727688 4969 scope.go:122] "RemoveContainer" containerID="6e3ddcd1b09643145e15f62e38a85c7bc88885d9c255ecb6112278381dcdf32a" Oct 31 01:15:15.734372 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:15.734347 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a880593-ae39-4bec-810a-ee2263ea9066" path="/var/lib/kubelet/pods/2a880593-ae39-4bec-810a-ee2263ea9066/volumes" Oct 31 01:15:16.265187 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:16.265096 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" event={"ID":"b76440b9-7df1-4a68-b594-574635feba46","Type":"ContainerStarted","Data":"844cc82e01e609dd6ee74f99926388d41a5c70866ca55114b167fe5526561737"} Oct 31 01:15:16.381305 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:16.381248 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" podStartSLOduration=16.381235272 podStartE2EDuration="16.381235272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:16.379733933 +0000 UTC m=+4348.855092355" watchObservedRunningTime="2026-10-31 01:15:16.381235272 +0000 UTC m=+4348.856593694" Oct 31 01:15:16.777320 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:16.777291 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl"] Oct 31 01:15:16.779796 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:16.779765 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-093220-zf9fl"] Oct 31 01:15:17.274335 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:17.274305 4969 generic.go:356] "Generic (PLEG): container finished" podID="b76440b9-7df1-4a68-b594-574635feba46" containerID="844cc82e01e609dd6ee74f99926388d41a5c70866ca55114b167fe5526561737" exitCode=60 Oct 31 01:15:17.274798 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:17.274374 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" event={"ID":"b76440b9-7df1-4a68-b594-574635feba46","Type":"ContainerDied","Data":"844cc82e01e609dd6ee74f99926388d41a5c70866ca55114b167fe5526561737"} Oct 31 01:15:17.274798 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:17.274431 4969 scope.go:122] "RemoveContainer" containerID="6e3ddcd1b09643145e15f62e38a85c7bc88885d9c255ecb6112278381dcdf32a" Oct 31 01:15:17.274798 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:17.274775 4969 scope.go:122] "RemoveContainer" containerID="844cc82e01e609dd6ee74f99926388d41a5c70866ca55114b167fe5526561737" Oct 31 01:15:17.275049 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:15:17.275027 4969 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-bqnpj_odh-ai-gateway-infra(b76440b9-7df1-4a68-b594-574635feba46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" podUID="b76440b9-7df1-4a68-b594-574635feba46" Oct 31 01:15:17.392730 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:17.392693 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj"] Oct 31 01:15:17.741768 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:17.741734 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7d2be3-f929-46db-8a57-4e0fd43350fe" path="/var/lib/kubelet/pods/7e7d2be3-f929-46db-8a57-4e0fd43350fe/volumes" Oct 31 01:15:18.424167 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:18.424142 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:18.569214 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:18.569142 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76440b9-7df1-4a68-b594-574635feba46-kube-api-access-s26wt\" (UniqueName: \"kubernetes.io/projected/b76440b9-7df1-4a68-b594-574635feba46-kube-api-access-s26wt\") pod \"b76440b9-7df1-4a68-b594-574635feba46\" (UID: \"b76440b9-7df1-4a68-b594-574635feba46\") " Oct 31 01:15:18.569214 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:18.569205 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76440b9-7df1-4a68-b594-574635feba46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76440b9-7df1-4a68-b594-574635feba46-maas-api-service-ca\") pod \"b76440b9-7df1-4a68-b594-574635feba46\" (UID: \"b76440b9-7df1-4a68-b594-574635feba46\") " Oct 31 01:15:18.569633 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:18.569605 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76440b9-7df1-4a68-b594-574635feba46-maas-api-service-ca" pod "b76440b9-7df1-4a68-b594-574635feba46" (UID: "b76440b9-7df1-4a68-b594-574635feba46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:18.571364 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:18.571343 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76440b9-7df1-4a68-b594-574635feba46-kube-api-access-s26wt" pod "b76440b9-7df1-4a68-b594-574635feba46" (UID: "b76440b9-7df1-4a68-b594-574635feba46"). InnerVolumeSpecName "kube-api-access-s26wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:18.671081 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:18.671053 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s26wt\" (UniqueName: \"kubernetes.io/projected/b76440b9-7df1-4a68-b594-574635feba46-kube-api-access-s26wt\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.671081 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:18.671079 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76440b9-7df1-4a68-b594-574635feba46-maas-api-service-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:19.289474 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:19.289446 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" Oct 31 01:15:19.289683 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:19.289440 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj" event={"ID":"b76440b9-7df1-4a68-b594-574635feba46","Type":"ContainerDied","Data":"e20e06ac6339edae109e1a35a28d1833179a73a9b5e9ec434e5902aed99886ea"} Oct 31 01:15:19.289683 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:19.289587 4969 scope.go:122] "RemoveContainer" containerID="844cc82e01e609dd6ee74f99926388d41a5c70866ca55114b167fe5526561737" Oct 31 01:15:19.453199 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:19.453170 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj"] Oct 31 01:15:19.502577 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:19.502548 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-bqnpj"] Oct 31 01:15:19.732645 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:19.732612 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76440b9-7df1-4a68-b594-574635feba46" path="/var/lib/kubelet/pods/b76440b9-7df1-4a68-b594-574635feba46/volumes" Oct 31 01:15:24.806497 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806460 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9"] Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806900 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b76440b9-7df1-4a68-b594-574635feba46" containerName="cleanup" Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806913 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76440b9-7df1-4a68-b594-574635feba46" containerName="cleanup" Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806934 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b76440b9-7df1-4a68-b594-574635feba46" containerName="cleanup" Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806939 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76440b9-7df1-4a68-b594-574635feba46" containerName="cleanup" Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806950 4969 cpu_manager.go:414] "containerMap: removing container" podUID="b76440b9-7df1-4a68-b594-574635feba46" containerName="cleanup" Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806955 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76440b9-7df1-4a68-b594-574635feba46" containerName="cleanup" Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806965 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2a880593-ae39-4bec-810a-ee2263ea9066" containerName="cleanup" Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806970 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a880593-ae39-4bec-810a-ee2263ea9066" containerName="cleanup" Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806978 4969 cpu_manager.go:414] "containerMap: removing container" podUID="2a880593-ae39-4bec-810a-ee2263ea9066" containerName="cleanup" Oct 31 01:15:24.807015 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.806984 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a880593-ae39-4bec-810a-ee2263ea9066" containerName="cleanup" Oct 31 01:15:24.807378 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.807057 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76440b9-7df1-4a68-b594-574635feba46" containerName="cleanup" Oct 31 01:15:24.807378 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.807066 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76440b9-7df1-4a68-b594-574635feba46" containerName="cleanup" Oct 31 01:15:24.807378 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.807072 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a880593-ae39-4bec-810a-ee2263ea9066" containerName="cleanup" Oct 31 01:15:24.807378 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.807077 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a880593-ae39-4bec-810a-ee2263ea9066" containerName="cleanup" Oct 31 01:15:24.807378 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.807086 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76440b9-7df1-4a68-b594-574635feba46" containerName="cleanup" Oct 31 01:15:24.811525 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.811508 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:24.864970 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.864941 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:24.914099 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.914064 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:15:24.916780 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.916757 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9"] Oct 31 01:15:24.928012 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.927984 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:24.928151 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.928094 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z69c\" (UniqueName: \"kubernetes.io/projected/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-kube-api-access-2z69c\") pod \"maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:24.928210 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:24.928179 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:25.029361 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:25.029329 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:25.029515 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:25.029383 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:25.029515 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:25.029419 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z69c\" (UniqueName: \"kubernetes.io/projected/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-kube-api-access-2z69c\") pod \"maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:25.030070 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:25.030046 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:25.032020 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:25.032001 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:25.090229 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:25.090170 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z69c\" (UniqueName: \"kubernetes.io/projected/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-kube-api-access-2z69c\") pod \"maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:25.125045 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:25.125015 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:25.501149 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:25.501114 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9"] Oct 31 01:15:25.502308 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:15:25.502281 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edaac0f_09ee_4a39_a6bb_d2d7146d1974.slice/crio-47d6784d9b0416ec5bb7581143ad2c00c70b00fc451025fc75418d7884649138 WatchSource:0}: Error finding container 47d6784d9b0416ec5bb7581143ad2c00c70b00fc451025fc75418d7884649138: Status 404 returned error can't find the container with id 47d6784d9b0416ec5bb7581143ad2c00c70b00fc451025fc75418d7884649138 Oct 31 01:15:26.340982 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:26.340941 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" event={"ID":"8edaac0f-09ee-4a39-a6bb-d2d7146d1974","Type":"ContainerStarted","Data":"0beb9147dd0c61e6c0dd6cecf91167de139d3cc6a4216b0ec514ced89a016f72"} Oct 31 01:15:26.341393 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:26.341006 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" event={"ID":"8edaac0f-09ee-4a39-a6bb-d2d7146d1974","Type":"ContainerStarted","Data":"47d6784d9b0416ec5bb7581143ad2c00c70b00fc451025fc75418d7884649138"} Oct 31 01:15:26.458120 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:26.458065 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" podStartSLOduration=2.458050706 podStartE2EDuration="2.458050706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:26.455372123 +0000 UTC m=+4358.930730546" watchObservedRunningTime="2026-10-31 01:15:26.458050706 +0000 UTC m=+4358.933409148" Oct 31 01:15:27.348444 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:27.348402 4969 generic.go:356] "Generic (PLEG): container finished" podID="8edaac0f-09ee-4a39-a6bb-d2d7146d1974" containerID="0beb9147dd0c61e6c0dd6cecf91167de139d3cc6a4216b0ec514ced89a016f72" exitCode=0 Oct 31 01:15:27.348871 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:27.348454 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" event={"ID":"8edaac0f-09ee-4a39-a6bb-d2d7146d1974","Type":"ContainerDied","Data":"0beb9147dd0c61e6c0dd6cecf91167de139d3cc6a4216b0ec514ced89a016f72"} Oct 31 01:15:28.498547 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.498523 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:28.666476 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.666449 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-openshift-ca\") pod \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " Oct 31 01:15:28.666698 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.666514 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-kube-api-access-2z69c\" (UniqueName: \"kubernetes.io/projected/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-kube-api-access-2z69c\") pod \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " Oct 31 01:15:28.666698 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.666635 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-xks-ca\") pod \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\" (UID: \"8edaac0f-09ee-4a39-a6bb-d2d7146d1974\") " Oct 31 01:15:28.666840 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.666820 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-openshift-ca" pod "8edaac0f-09ee-4a39-a6bb-d2d7146d1974" (UID: "8edaac0f-09ee-4a39-a6bb-d2d7146d1974"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:28.666931 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.666911 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:28.668888 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.668867 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-kube-api-access-2z69c" pod "8edaac0f-09ee-4a39-a6bb-d2d7146d1974" (UID: "8edaac0f-09ee-4a39-a6bb-d2d7146d1974"). InnerVolumeSpecName "kube-api-access-2z69c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:28.668987 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.668973 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-xks-ca" pod "8edaac0f-09ee-4a39-a6bb-d2d7146d1974" (UID: "8edaac0f-09ee-4a39-a6bb-d2d7146d1974"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:28.767475 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.767446 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z69c\" (UniqueName: \"kubernetes.io/projected/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-kube-api-access-2z69c\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:28.767475 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:28.767472 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8edaac0f-09ee-4a39-a6bb-d2d7146d1974-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:29.192779 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:29.192745 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj"] Oct 31 01:15:29.243860 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:29.243822 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s"] Oct 31 01:15:29.362566 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:29.362534 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" Oct 31 01:15:29.362752 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:29.362581 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9" event={"ID":"8edaac0f-09ee-4a39-a6bb-d2d7146d1974","Type":"ContainerDied","Data":"47d6784d9b0416ec5bb7581143ad2c00c70b00fc451025fc75418d7884649138"} Oct 31 01:15:29.362752 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:29.362637 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47d6784d9b0416ec5bb7581143ad2c00c70b00fc451025fc75418d7884649138" Oct 31 01:15:32.726934 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:15:32.726893 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:15:35.757010 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.756976 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2"] Oct 31 01:15:35.757439 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.757425 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8edaac0f-09ee-4a39-a6bb-d2d7146d1974" containerName="revoke-keys" Oct 31 01:15:35.757491 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.757441 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edaac0f-09ee-4a39-a6bb-d2d7146d1974" containerName="revoke-keys" Oct 31 01:15:35.757552 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.757542 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edaac0f-09ee-4a39-a6bb-d2d7146d1974" containerName="revoke-keys" Oct 31 01:15:35.760871 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.760854 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:35.818981 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.818947 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2"] Oct 31 01:15:35.865157 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.865122 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:35.865328 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.865204 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bg45c\"" Oct 31 01:15:35.933715 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.933676 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff441ee3-fmvb2\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:35.933907 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.933790 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff441ee3-fmvb2\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:35.933907 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:35.933835 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm2j5\" (UniqueName: \"kubernetes.io/projected/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-kube-api-access-mm2j5\") pod \"maas-api-revoke-sub-ff441ee3-fmvb2\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:36.034830 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.034744 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff441ee3-fmvb2\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:36.034830 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.034811 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff441ee3-fmvb2\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:36.035036 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.034835 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2j5\" (UniqueName: \"kubernetes.io/projected/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-kube-api-access-mm2j5\") pod \"maas-api-revoke-sub-ff441ee3-fmvb2\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:36.035465 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.035438 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff441ee3-fmvb2\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:36.037281 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.037261 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff441ee3-fmvb2\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:36.095952 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.095924 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2j5\" (UniqueName: \"kubernetes.io/projected/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-kube-api-access-mm2j5\") pod \"maas-api-revoke-sub-ff441ee3-fmvb2\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:36.279338 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.279295 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j"] Oct 31 01:15:36.373351 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.373322 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:36.381961 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.381931 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd"] Oct 31 01:15:36.755682 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:36.755649 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2"] Oct 31 01:15:36.757478 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:15:36.757449 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e055d03_71d0_4d2d_a5dd_5735cdcc9a33.slice/crio-f6b137e34010e26b9dbce073898eb7ac68cbb3cde04ff816a925d7cfe163806e WatchSource:0}: Error finding container f6b137e34010e26b9dbce073898eb7ac68cbb3cde04ff816a925d7cfe163806e: Status 404 returned error can't find the container with id f6b137e34010e26b9dbce073898eb7ac68cbb3cde04ff816a925d7cfe163806e Oct 31 01:15:37.414857 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:37.414816 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" event={"ID":"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33","Type":"ContainerStarted","Data":"e6021a99ef8362778cf133b1a8540225c902828d40d3856ef3d6b2600d597cfc"} Oct 31 01:15:37.415073 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:37.414915 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" event={"ID":"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33","Type":"ContainerStarted","Data":"f6b137e34010e26b9dbce073898eb7ac68cbb3cde04ff816a925d7cfe163806e"} Oct 31 01:15:37.531261 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:37.531200 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" podStartSLOduration=2.5311855100000002 podStartE2EDuration="2.53118551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:37.529140276 +0000 UTC m=+4370.004498718" watchObservedRunningTime="2026-10-31 01:15:37.53118551 +0000 UTC m=+4370.006543932" Oct 31 01:15:38.423489 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:38.423446 4969 generic.go:356] "Generic (PLEG): container finished" podID="8e055d03-71d0-4d2d-a5dd-5735cdcc9a33" containerID="e6021a99ef8362778cf133b1a8540225c902828d40d3856ef3d6b2600d597cfc" exitCode=0 Oct 31 01:15:38.423983 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:38.423534 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" event={"ID":"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33","Type":"ContainerDied","Data":"e6021a99ef8362778cf133b1a8540225c902828d40d3856ef3d6b2600d597cfc"} Oct 31 01:15:39.572479 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.572455 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:39.661403 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.661375 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-openshift-ca\") pod \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " Oct 31 01:15:39.661607 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.661433 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-kube-api-access-mm2j5\" (UniqueName: \"kubernetes.io/projected/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-kube-api-access-mm2j5\") pod \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " Oct 31 01:15:39.661607 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.661580 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-xks-ca\") pod \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\" (UID: \"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33\") " Oct 31 01:15:39.661866 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.661825 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-openshift-ca" pod "8e055d03-71d0-4d2d-a5dd-5735cdcc9a33" (UID: "8e055d03-71d0-4d2d-a5dd-5735cdcc9a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:39.661986 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.661927 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-openshift-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.663826 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.663803 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-xks-ca" pod "8e055d03-71d0-4d2d-a5dd-5735cdcc9a33" (UID: "8e055d03-71d0-4d2d-a5dd-5735cdcc9a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:39.663946 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.663926 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-kube-api-access-mm2j5" pod "8e055d03-71d0-4d2d-a5dd-5735cdcc9a33" (UID: "8e055d03-71d0-4d2d-a5dd-5735cdcc9a33"). InnerVolumeSpecName "kube-api-access-mm2j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.763173 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.763145 4969 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-maas-api-xks-ca\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.763299 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:39.763201 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm2j5\" (UniqueName: \"kubernetes.io/projected/8e055d03-71d0-4d2d-a5dd-5735cdcc9a33-kube-api-access-mm2j5\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:40.437635 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:40.437602 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" Oct 31 01:15:40.437816 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:40.437635 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff441ee3-fmvb2" event={"ID":"8e055d03-71d0-4d2d-a5dd-5735cdcc9a33","Type":"ContainerDied","Data":"f6b137e34010e26b9dbce073898eb7ac68cbb3cde04ff816a925d7cfe163806e"} Oct 31 01:15:40.437816 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:40.437685 4969 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b137e34010e26b9dbce073898eb7ac68cbb3cde04ff816a925d7cfe163806e" Oct 31 01:15:41.279885 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:41.279845 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" podUID="bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" containerName="payload-processing" containerID="cri-o://db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735" gracePeriod=25 Oct 31 01:15:41.383183 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:41.383143 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" podUID="890b0974-c3c4-4d27-a1b8-fb3f018b14dc" containerName="payload-pre-processing" containerID="cri-o://953ead5d0d4f2ddbec71025ca8907c89aab07b6bbaeceb30e6e17d393cc5bf40" gracePeriod=25 Oct 31 01:15:44.213190 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.213130 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" containerID="cri-o://e4c5bfd4da6ef2501b622bf95949fb9dd30d3df15c9c9744d9ab3db493ccecb3" gracePeriod=45 Oct 31 01:15:44.263576 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.263542 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" containerID="cri-o://6c6048760df7f81d9a2f5baa9e58449d6676018b7f93eb62e07436faa3a20716" gracePeriod=45 Oct 31 01:15:44.419951 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.419897 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": dial tcp 10.133.0.99:8000: connect: connection refused" Oct 31 01:15:44.473034 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.472963 4969 generic.go:356] "Generic (PLEG): container finished" podID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerID="6c6048760df7f81d9a2f5baa9e58449d6676018b7f93eb62e07436faa3a20716" exitCode=0 Oct 31 01:15:44.473172 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.473032 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" event={"ID":"a618e25b-20c7-4424-a7ac-3d5a89d9a24a","Type":"ContainerDied","Data":"6c6048760df7f81d9a2f5baa9e58449d6676018b7f93eb62e07436faa3a20716"} Oct 31 01:15:44.475149 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.475119 4969 generic.go:356] "Generic (PLEG): container finished" podID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerID="e4c5bfd4da6ef2501b622bf95949fb9dd30d3df15c9c9744d9ab3db493ccecb3" exitCode=0 Oct 31 01:15:44.475289 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.475194 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" event={"ID":"00e200c4-beab-4ff6-a5cb-c2864b4dfe31","Type":"ContainerDied","Data":"e4c5bfd4da6ef2501b622bf95949fb9dd30d3df15c9c9744d9ab3db493ccecb3"} Oct 31 01:15:44.529149 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.529126 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 01:15:44.532409 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.532389 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 01:15:44.607887 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.607858 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-home\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-home\") pod \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " Oct 31 01:15:44.608041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.607894 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tls-certs\" (UniqueName: \"kubernetes.io/secret/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tls-certs\") pod \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " Oct 31 01:15:44.608041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.607922 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tmp-dir\") pod \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " Oct 31 01:15:44.608041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.607954 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-home\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-home\") pod \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " Oct 31 01:15:44.608041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.607987 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-kube-api-access-wmrq7\" (UniqueName: \"kubernetes.io/projected/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-kube-api-access-wmrq7\") pod \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " Oct 31 01:15:44.608041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608040 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-model-cache\") pod \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " Oct 31 01:15:44.608324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608064 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tls-certs\") pod \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " Oct 31 01:15:44.608324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608099 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-model-cache\") pod \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " Oct 31 01:15:44.608324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608150 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-dshm\") pod \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " Oct 31 01:15:44.608324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608150 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-home" pod "a618e25b-20c7-4424-a7ac-3d5a89d9a24a" (UID: "a618e25b-20c7-4424-a7ac-3d5a89d9a24a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.608324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608222 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tmp-dir" pod "00e200c4-beab-4ff6-a5cb-c2864b4dfe31" (UID: "00e200c4-beab-4ff6-a5cb-c2864b4dfe31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.608324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608228 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-dshm\") pod \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " Oct 31 01:15:44.608324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608245 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-home" pod "00e200c4-beab-4ff6-a5cb-c2864b4dfe31" (UID: "00e200c4-beab-4ff6-a5cb-c2864b4dfe31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.608324 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608316 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-model-cache" pod "a618e25b-20c7-4424-a7ac-3d5a89d9a24a" (UID: "a618e25b-20c7-4424-a7ac-3d5a89d9a24a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.608820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608333 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-kube-api-access-6bhrg\" (UniqueName: \"kubernetes.io/projected/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-kube-api-access-6bhrg\") pod \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\" (UID: \"00e200c4-beab-4ff6-a5cb-c2864b4dfe31\") " Oct 31 01:15:44.608820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608366 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tmp-dir\") pod \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\" (UID: \"a618e25b-20c7-4424-a7ac-3d5a89d9a24a\") " Oct 31 01:15:44.608820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608505 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-model-cache" pod "00e200c4-beab-4ff6-a5cb-c2864b4dfe31" (UID: "00e200c4-beab-4ff6-a5cb-c2864b4dfe31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.608820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608712 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tmp-dir" pod "a618e25b-20c7-4424-a7ac-3d5a89d9a24a" (UID: "a618e25b-20c7-4424-a7ac-3d5a89d9a24a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.608820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608810 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.609275 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608826 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.609275 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608841 4969 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-home\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.609275 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608854 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.609275 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608867 4969 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-model-cache\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.609275 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.608889 4969 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tmp-dir\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.610736 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.610716 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-dshm" pod "00e200c4-beab-4ff6-a5cb-c2864b4dfe31" (UID: "00e200c4-beab-4ff6-a5cb-c2864b4dfe31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.610847 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.610795 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-kube-api-access-6bhrg" pod "00e200c4-beab-4ff6-a5cb-c2864b4dfe31" (UID: "00e200c4-beab-4ff6-a5cb-c2864b4dfe31"). InnerVolumeSpecName "kube-api-access-6bhrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:44.610899 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.610880 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tls-certs" pod "00e200c4-beab-4ff6-a5cb-c2864b4dfe31" (UID: "00e200c4-beab-4ff6-a5cb-c2864b4dfe31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:44.610936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.610910 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-kube-api-access-wmrq7" pod "a618e25b-20c7-4424-a7ac-3d5a89d9a24a" (UID: "a618e25b-20c7-4424-a7ac-3d5a89d9a24a"). InnerVolumeSpecName "kube-api-access-wmrq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:44.611292 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.611276 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tls-certs" pod "a618e25b-20c7-4424-a7ac-3d5a89d9a24a" (UID: "a618e25b-20c7-4424-a7ac-3d5a89d9a24a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:44.611334 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.611299 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-dshm" pod "a618e25b-20c7-4424-a7ac-3d5a89d9a24a" (UID: "a618e25b-20c7-4424-a7ac-3d5a89d9a24a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:44.709703 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.709671 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.709703 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.709705 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bhrg\" (UniqueName: \"kubernetes.io/projected/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-kube-api-access-6bhrg\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.709910 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.709722 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e200c4-beab-4ff6-a5cb-c2864b4dfe31-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.709910 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.709736 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrq7\" (UniqueName: \"kubernetes.io/projected/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-kube-api-access-wmrq7\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.709910 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.709750 4969 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-tls-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:44.709910 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:44.709764 4969 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a618e25b-20c7-4424-a7ac-3d5a89d9a24a-dshm\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:45.483550 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.483469 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" Oct 31 01:15:45.483550 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.483513 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s" event={"ID":"a618e25b-20c7-4424-a7ac-3d5a89d9a24a","Type":"ContainerDied","Data":"eb687b9ae688240937eba8b289b1f3ba8d0d38efec40c0801992bcce643a660f"} Oct 31 01:15:45.484279 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.483571 4969 scope.go:122] "RemoveContainer" containerID="6c6048760df7f81d9a2f5baa9e58449d6676018b7f93eb62e07436faa3a20716" Oct 31 01:15:45.485297 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.485278 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" Oct 31 01:15:45.485405 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.485322 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj" event={"ID":"00e200c4-beab-4ff6-a5cb-c2864b4dfe31","Type":"ContainerDied","Data":"5d9166610dfdca0e34b20498f43293c6b3922e5251e378ae3b39f7c88e484a1a"} Oct 31 01:15:45.494012 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.493998 4969 scope.go:122] "RemoveContainer" containerID="e4c5bfd4da6ef2501b622bf95949fb9dd30d3df15c9c9744d9ab3db493ccecb3" Oct 31 01:15:45.646987 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.646952 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s"] Oct 31 01:15:45.651217 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.651193 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-w4-0144bd-kserve-668949c45b-rlq8s"] Oct 31 01:15:45.734477 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.734391 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" path="/var/lib/kubelet/pods/a618e25b-20c7-4424-a7ac-3d5a89d9a24a/volumes" Oct 31 01:15:45.760323 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.760294 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj"] Oct 31 01:15:45.763391 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:45.763366 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0144bd/distinct-2-w4-0144bd-kserve-86bd7fb8bc-l7nnj"] Oct 31 01:15:47.733991 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:47.733955 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" path="/var/lib/kubelet/pods/00e200c4-beab-4ff6-a5cb-c2864b4dfe31/volumes" Oct 31 01:15:49.097545 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:49.097509 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" podUID="bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:49.097545 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:49.097537 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" podUID="890b0974-c3c4-4d27-a1b8-fb3f018b14dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:52.567371 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:52.566317 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9"] Oct 31 01:15:52.578586 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:52.578556 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0144bd-zc5n9"] Oct 31 01:15:53.733690 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:53.733649 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8edaac0f-09ee-4a39-a6bb-d2d7146d1974" path="/var/lib/kubelet/pods/8edaac0f-09ee-4a39-a6bb-d2d7146d1974/volumes" Oct 31 01:15:59.096951 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:59.096910 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" podUID="bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:59.097423 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:59.096910 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" podUID="890b0974-c3c4-4d27-a1b8-fb3f018b14dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:59.973518 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:59.973485 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs"] Oct 31 01:15:59.973794 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:15:59.973742 4969 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" containerID="cri-o://993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b" gracePeriod=30 Oct 31 01:16:01.515856 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.515821 4969 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:16:01.516282 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.515890 4969 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:16:01.539483 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.539413 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 01:16:01.594814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.594779 4969 generic.go:356] "Generic (PLEG): container finished" podID="890b0974-c3c4-4d27-a1b8-fb3f018b14dc" containerID="953ead5d0d4f2ddbec71025ca8907c89aab07b6bbaeceb30e6e17d393cc5bf40" exitCode=0 Oct 31 01:16:01.595000 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.594849 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" event={"ID":"890b0974-c3c4-4d27-a1b8-fb3f018b14dc","Type":"ContainerDied","Data":"953ead5d0d4f2ddbec71025ca8907c89aab07b6bbaeceb30e6e17d393cc5bf40"} Oct 31 01:16:01.596493 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.596466 4969 generic.go:356] "Generic (PLEG): container finished" podID="bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" containerID="db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735" exitCode=0 Oct 31 01:16:01.596657 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.596535 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" Oct 31 01:16:01.596657 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.596552 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" event={"ID":"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4","Type":"ContainerDied","Data":"db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735"} Oct 31 01:16:01.596657 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.596640 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j" event={"ID":"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4","Type":"ContainerDied","Data":"e8b40ca33473e44dee10896ec70ec90d378fe51708be56a40c74ad6d751e30e4"} Oct 31 01:16:01.596838 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.596672 4969 scope.go:122] "RemoveContainer" containerID="db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735" Oct 31 01:16:01.607716 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.607694 4969 scope.go:122] "RemoveContainer" containerID="db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735" Oct 31 01:16:01.608037 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:16:01.608017 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735\": container with ID starting with db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735 not found: ID does not exist" containerID="db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735" Oct 31 01:16:01.608113 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.608046 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735"} err="failed to get container status \"db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735\": rpc error: code = NotFound desc = could not find container \"db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735\": container with ID starting with db342a94bd4aa51f82450bd1118a865612805d806bb952889aadc558de29a735 not found: ID does not exist" Oct 31 01:16:01.636855 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.636829 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 01:16:01.664512 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.664479 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-kube-api-access-s9sfb\" (UniqueName: \"kubernetes.io/projected/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-kube-api-access-s9sfb\") pod \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\" (UID: \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\") " Oct 31 01:16:01.664718 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.664662 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-plugins-config-volume\") pod \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\" (UID: \"bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4\") " Oct 31 01:16:01.665089 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.665062 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-plugins-config-volume" pod "bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" (UID: "bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:01.667005 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.666980 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-kube-api-access-s9sfb" pod "bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" (UID: "bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4"). InnerVolumeSpecName "kube-api-access-s9sfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:01.765497 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.765461 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-kube-api-access-pgnzj\" (UniqueName: \"kubernetes.io/projected/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-kube-api-access-pgnzj\") pod \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\" (UID: \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\") " Oct 31 01:16:01.765713 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.765522 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-plugins-config-volume\") pod \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\" (UID: \"890b0974-c3c4-4d27-a1b8-fb3f018b14dc\") " Oct 31 01:16:01.765812 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.765797 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:01.765883 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.765815 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9sfb\" (UniqueName: \"kubernetes.io/projected/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4-kube-api-access-s9sfb\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:01.766013 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.765979 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-plugins-config-volume" pod "890b0974-c3c4-4d27-a1b8-fb3f018b14dc" (UID: "890b0974-c3c4-4d27-a1b8-fb3f018b14dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:01.767820 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.767800 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-kube-api-access-pgnzj" pod "890b0974-c3c4-4d27-a1b8-fb3f018b14dc" (UID: "890b0974-c3c4-4d27-a1b8-fb3f018b14dc"). InnerVolumeSpecName "kube-api-access-pgnzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:01.867092 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.867065 4969 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-plugins-config-volume\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:01.867092 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.867091 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgnzj\" (UniqueName: \"kubernetes.io/projected/890b0974-c3c4-4d27-a1b8-fb3f018b14dc-kube-api-access-pgnzj\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:01.973144 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:01.973109 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b5nfc_40085cb1-46da-478b-bf95-6da06fd5e85f/manager/0.log" Oct 31 01:16:02.011674 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:02.011638 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j"] Oct 31 01:16:02.017537 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:02.017513 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0144bd-546dccbd5c-tnr5j"] Oct 31 01:16:02.294936 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:02.294909 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zsk4m_7e3ba2f7-5e86-496b-82fb-956ac8db9d60/manager/0.log" Oct 31 01:16:02.528453 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:02.528421 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dnvj9_64c1ce0f-d843-4306-8d6f-8c37a72ec455/kuadrant-console-plugin/0.log" Oct 31 01:16:02.603457 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:02.603366 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" Oct 31 01:16:02.603671 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:02.603366 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd" event={"ID":"890b0974-c3c4-4d27-a1b8-fb3f018b14dc","Type":"ContainerDied","Data":"3f6221a0ebccbc2ed74b0412e9b1c8cc5da5f516ed942e42b0b3c52f580b823c"} Oct 31 01:16:02.603671 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:02.603523 4969 scope.go:122] "RemoveContainer" containerID="953ead5d0d4f2ddbec71025ca8907c89aab07b6bbaeceb30e6e17d393cc5bf40" Oct 31 01:16:02.726704 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:16:02.726667 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:16:02.767962 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:02.767931 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd"] Oct 31 01:16:02.817991 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:02.817948 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0144bd-c955f6b6c-4f6nd"] Oct 31 01:16:03.740138 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:03.740103 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890b0974-c3c4-4d27-a1b8-fb3f018b14dc" path="/var/lib/kubelet/pods/890b0974-c3c4-4d27-a1b8-fb3f018b14dc/volumes" Oct 31 01:16:03.740538 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:03.740499 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" path="/var/lib/kubelet/pods/bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4/volumes" Oct 31 01:16:05.228297 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.228272 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 01:16:05.300855 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.300785 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-token\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-token\") pod \"6c0668f1-3bb4-4cad-976e-9b97244f5622\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " Oct 31 01:16:05.300855 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.300840 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-kube-api-access-bfmlh\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-kube-api-access-bfmlh\") pod \"6c0668f1-3bb4-4cad-976e-9b97244f5622\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " Oct 31 01:16:05.301066 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.300862 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-certs\") pod \"6c0668f1-3bb4-4cad-976e-9b97244f5622\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " Oct 31 01:16:05.301066 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.300976 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-data\") pod \"6c0668f1-3bb4-4cad-976e-9b97244f5622\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " Oct 31 01:16:05.301066 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301027 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0668f1-3bb4-4cad-976e-9b97244f5622-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0668f1-3bb4-4cad-976e-9b97244f5622-istiod-ca-cert\") pod \"6c0668f1-3bb4-4cad-976e-9b97244f5622\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " Oct 31 01:16:05.301253 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301120 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-socket\") pod \"6c0668f1-3bb4-4cad-976e-9b97244f5622\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " Oct 31 01:16:05.301253 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301130 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-certs" pod "6c0668f1-3bb4-4cad-976e-9b97244f5622" (UID: "6c0668f1-3bb4-4cad-976e-9b97244f5622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:05.301253 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301168 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-podinfo\") pod \"6c0668f1-3bb4-4cad-976e-9b97244f5622\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " Oct 31 01:16:05.301253 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301243 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-credential-socket\") pod \"6c0668f1-3bb4-4cad-976e-9b97244f5622\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " Oct 31 01:16:05.301492 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301264 4969 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-envoy\") pod \"6c0668f1-3bb4-4cad-976e-9b97244f5622\" (UID: \"6c0668f1-3bb4-4cad-976e-9b97244f5622\") " Oct 31 01:16:05.301492 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301268 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-data" pod "6c0668f1-3bb4-4cad-976e-9b97244f5622" (UID: "6c0668f1-3bb4-4cad-976e-9b97244f5622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:05.301492 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301344 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-socket" pod "6c0668f1-3bb4-4cad-976e-9b97244f5622" (UID: "6c0668f1-3bb4-4cad-976e-9b97244f5622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:05.301672 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301501 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-credential-socket" pod "6c0668f1-3bb4-4cad-976e-9b97244f5622" (UID: "6c0668f1-3bb4-4cad-976e-9b97244f5622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:05.301672 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301522 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0668f1-3bb4-4cad-976e-9b97244f5622-istiod-ca-cert" pod "6c0668f1-3bb4-4cad-976e-9b97244f5622" (UID: "6c0668f1-3bb4-4cad-976e-9b97244f5622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:05.301672 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301654 4969 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-certs\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:05.301819 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301676 4969 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-data\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:05.301819 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301690 4969 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0668f1-3bb4-4cad-976e-9b97244f5622-istiod-ca-cert\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:05.301819 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301705 4969 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-workload-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:05.301819 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.301721 4969 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-credential-socket\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:05.303353 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.303330 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-token" pod "6c0668f1-3bb4-4cad-976e-9b97244f5622" (UID: "6c0668f1-3bb4-4cad-976e-9b97244f5622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:05.303486 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.303350 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-kube-api-access-bfmlh" pod "6c0668f1-3bb4-4cad-976e-9b97244f5622" (UID: "6c0668f1-3bb4-4cad-976e-9b97244f5622"). InnerVolumeSpecName "kube-api-access-bfmlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:05.303554 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.303525 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-podinfo" pod "6c0668f1-3bb4-4cad-976e-9b97244f5622" (UID: "6c0668f1-3bb4-4cad-976e-9b97244f5622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:16:05.303732 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.303718 4969 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-envoy" pod "6c0668f1-3bb4-4cad-976e-9b97244f5622" (UID: "6c0668f1-3bb4-4cad-976e-9b97244f5622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:16:05.402928 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.402892 4969 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-envoy\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:05.402928 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.402920 4969 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-token\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:05.402928 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.402930 4969 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfmlh\" (UniqueName: \"kubernetes.io/projected/6c0668f1-3bb4-4cad-976e-9b97244f5622-kube-api-access-bfmlh\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:05.403171 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.402939 4969 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0668f1-3bb4-4cad-976e-9b97244f5622-istio-podinfo\") on node \"ip-10-0-141-63.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:16:05.625173 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.625141 4969 generic.go:356] "Generic (PLEG): container finished" podID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerID="993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b" exitCode=0 Oct 31 01:16:05.625354 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.625190 4969 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" Oct 31 01:16:05.625354 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.625228 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" event={"ID":"6c0668f1-3bb4-4cad-976e-9b97244f5622","Type":"ContainerDied","Data":"993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b"} Oct 31 01:16:05.625354 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.625276 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs" event={"ID":"6c0668f1-3bb4-4cad-976e-9b97244f5622","Type":"ContainerDied","Data":"a75f139102b79b6fb092c333d9266b37f56c7c9b0ecaf4a66377bf6f7f98da16"} Oct 31 01:16:05.625354 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.625300 4969 scope.go:122] "RemoveContainer" containerID="993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b" Oct 31 01:16:05.636414 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.636393 4969 scope.go:122] "RemoveContainer" containerID="8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708" Oct 31 01:16:05.645233 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.645210 4969 scope.go:122] "RemoveContainer" containerID="993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b" Oct 31 01:16:05.645634 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:16:05.645605 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b\": container with ID starting with 993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b not found: ID does not exist" containerID="993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b" Oct 31 01:16:05.645760 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.645649 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b"} err="failed to get container status \"993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b\": rpc error: code = NotFound desc = could not find container \"993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b\": container with ID starting with 993a028a19534cf0bc30c23194fa4d07a4001ac63e295fb19441d08166ec028b not found: ID does not exist" Oct 31 01:16:05.645760 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.645674 4969 scope.go:122] "RemoveContainer" containerID="8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708" Oct 31 01:16:05.647872 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:16:05.647848 4969 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708\": container with ID starting with 8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708 not found: ID does not exist" containerID="8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708" Oct 31 01:16:05.647970 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.647875 4969 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708"} err="failed to get container status \"8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708\": rpc error: code = NotFound desc = could not find container \"8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708\": container with ID starting with 8ee0ea13bad57b767ae430415ee647d619ca2ffbab1096bad69a729f8699c708 not found: ID does not exist" Oct 31 01:16:05.790758 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.790727 4969 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs"] Oct 31 01:16:05.794540 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:05.794509 4969 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-z4wxs"] Oct 31 01:16:07.286815 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:07.286783 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jgwjj_fb9cbf01-d844-41ce-8778-56c3d92f9862/server/0.log" Oct 31 01:16:07.733555 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:07.733514 4969 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" path="/var/lib/kubelet/pods/6c0668f1-3bb4-4cad-976e-9b97244f5622/volumes" Oct 31 01:16:07.810764 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:07.810732 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-svrwj_4ce42331-4673-4ade-add7-cc6a77a308ec/proxy/0.log" Oct 31 01:16:21.599308 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:21.599266 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jgwjj_fb9cbf01-d844-41ce-8778-56c3d92f9862/server/0.log" Oct 31 01:16:22.128175 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:22.128140 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-svrwj_4ce42331-4673-4ade-add7-cc6a77a308ec/proxy/0.log" Oct 31 01:16:24.850379 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:24.850349 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ff441ee3-fmvb2_8e055d03-71d0-4d2d-a5dd-5735cdcc9a33/revoke-keys/0.log" Oct 31 01:16:25.080161 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:25.080121 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4ld6r_b522953e-42b0-4a9e-82db-145e50b8abe3/postgres/0.log" Oct 31 01:16:27.092992 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:27.092959 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b5nfc_40085cb1-46da-478b-bf95-6da06fd5e85f/manager/0.log" Oct 31 01:16:27.398806 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:27.398769 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zsk4m_7e3ba2f7-5e86-496b-82fb-956ac8db9d60/manager/0.log" Oct 31 01:16:27.626538 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:27.626508 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dnvj9_64c1ce0f-d843-4306-8d6f-8c37a72ec455/kuadrant-console-plugin/0.log" Oct 31 01:16:30.144264 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:30.144232 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6_3dad313d-11a2-440b-a9f9-c0509d6bbcc6/istio-proxy/0.log" Oct 31 01:16:32.177369 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:32.177336 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94bcddf7f-vbt59_2b251574-895b-4ac8-9688-3e094921637a/router/0.log" Oct 31 01:16:32.726123 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:16:32.726088 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:16:33.093091 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:33.093010 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-ljs8d_5337c97b-b317-4aaf-a6fd-a94fa2466175/main/0.log" Oct 31 01:16:33.650061 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:33.650032 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-5qn4x_2bd6f9fd-aafe-4a92-ac8c-53a0944d2b52/main/0.log" Oct 31 01:16:34.205008 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:34.204974 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w9gck_783e7d8f-b5bd-4ef4-8dbd-d3f3309e9b3d/main/0.log" Oct 31 01:16:35.062536 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:35.062509 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-59cxw_f8736325-6dcb-4b95-ac2c-a9a25635af0a/main/0.log" Oct 31 01:16:45.435101 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435067 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7kk4f/must-gather-c4fs5"] Oct 31 01:16:45.435516 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435444 4969 cpu_manager.go:414] "containerMap: removing container" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" Oct 31 01:16:45.435516 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435459 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" Oct 31 01:16:45.435516 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435486 4969 cpu_manager.go:414] "containerMap: removing container" podUID="bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" containerName="payload-processing" Oct 31 01:16:45.435516 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435491 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" containerName="payload-processing" Oct 31 01:16:45.435516 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435500 4969 cpu_manager.go:414] "containerMap: removing container" podUID="8e055d03-71d0-4d2d-a5dd-5735cdcc9a33" containerName="revoke-keys" Oct 31 01:16:45.435516 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435506 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e055d03-71d0-4d2d-a5dd-5735cdcc9a33" containerName="revoke-keys" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435522 4969 cpu_manager.go:414] "containerMap: removing container" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435527 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435536 4969 cpu_manager.go:414] "containerMap: removing container" podUID="890b0974-c3c4-4d27-a1b8-fb3f018b14dc" containerName="payload-pre-processing" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435542 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890b0974-c3c4-4d27-a1b8-fb3f018b14dc" containerName="payload-pre-processing" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435550 4969 cpu_manager.go:414] "containerMap: removing container" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435554 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435562 4969 cpu_manager.go:414] "containerMap: removing container" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435568 4969 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435657 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2d7e43-d4ba-4d8e-8a37-80b52b5e3bc4" containerName="payload-processing" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435668 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e055d03-71d0-4d2d-a5dd-5735cdcc9a33" containerName="revoke-keys" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435680 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435688 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="890b0974-c3c4-4d27-a1b8-fb3f018b14dc" containerName="payload-pre-processing" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435697 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e200c4-beab-4ff6-a5cb-c2864b4dfe31" containerName="main" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435706 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0668f1-3bb4-4cad-976e-9b97244f5622" containerName="istio-proxy" Oct 31 01:16:45.435814 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.435711 4969 memory_manager.go:365] "RemoveStaleState removing state" podUID="a618e25b-20c7-4424-a7ac-3d5a89d9a24a" containerName="main" Oct 31 01:16:45.439207 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.439184 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7kk4f/must-gather-c4fs5" Oct 31 01:16:45.491382 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.491356 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7kk4f\"/\"default-dockercfg-9t2nr\"" Oct 31 01:16:45.540661 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.540625 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7kk4f\"/\"openshift-service-ca.crt\"" Oct 31 01:16:45.540821 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.540666 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7kk4f\"/\"kube-root-ca.crt\"" Oct 31 01:16:45.543304 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.543288 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7kk4f/must-gather-c4fs5"] Oct 31 01:16:45.545373 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.545354 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m9n5w/must-gather-cjcfd"] Oct 31 01:16:45.549043 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.549024 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m9n5w/must-gather-cjcfd" Oct 31 01:16:45.550349 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.550328 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4cb51f18-afa0-4067-bb42-729af3290db1-must-gather-output\") pod \"must-gather-c4fs5\" (UID: \"4cb51f18-afa0-4067-bb42-729af3290db1\") " pod="openshift-must-gather-7kk4f/must-gather-c4fs5" Oct 31 01:16:45.550470 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.550450 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h546x\" (UniqueName: \"kubernetes.io/projected/4cb51f18-afa0-4067-bb42-729af3290db1-kube-api-access-h546x\") pod \"must-gather-c4fs5\" (UID: \"4cb51f18-afa0-4067-bb42-729af3290db1\") " pod="openshift-must-gather-7kk4f/must-gather-c4fs5" Oct 31 01:16:45.588947 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.588923 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m9n5w/must-gather-cjcfd"] Oct 31 01:16:45.601122 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.601099 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m9n5w\"/\"openshift-service-ca.crt\"" Oct 31 01:16:45.601237 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.601154 4969 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m9n5w\"/\"kube-root-ca.crt\"" Oct 31 01:16:45.601443 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.601425 4969 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m9n5w\"/\"default-dockercfg-2hfcm\"" Oct 31 01:16:45.651941 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.651917 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4cb51f18-afa0-4067-bb42-729af3290db1-must-gather-output\") pod \"must-gather-c4fs5\" (UID: \"4cb51f18-afa0-4067-bb42-729af3290db1\") " pod="openshift-must-gather-7kk4f/must-gather-c4fs5" Oct 31 01:16:45.652044 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.651970 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpdm\" (UniqueName: \"kubernetes.io/projected/29f5d09c-97b3-4ad7-af24-04464a0da0d1-kube-api-access-8lpdm\") pod \"must-gather-cjcfd\" (UID: \"29f5d09c-97b3-4ad7-af24-04464a0da0d1\") " pod="openshift-must-gather-m9n5w/must-gather-cjcfd" Oct 31 01:16:45.652044 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.652000 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h546x\" (UniqueName: \"kubernetes.io/projected/4cb51f18-afa0-4067-bb42-729af3290db1-kube-api-access-h546x\") pod \"must-gather-c4fs5\" (UID: \"4cb51f18-afa0-4067-bb42-729af3290db1\") " pod="openshift-must-gather-7kk4f/must-gather-c4fs5" Oct 31 01:16:45.652122 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.652041 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29f5d09c-97b3-4ad7-af24-04464a0da0d1-must-gather-output\") pod \"must-gather-cjcfd\" (UID: \"29f5d09c-97b3-4ad7-af24-04464a0da0d1\") " pod="openshift-must-gather-m9n5w/must-gather-cjcfd" Oct 31 01:16:45.652288 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.652268 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4cb51f18-afa0-4067-bb42-729af3290db1-must-gather-output\") pod \"must-gather-c4fs5\" (UID: \"4cb51f18-afa0-4067-bb42-729af3290db1\") " pod="openshift-must-gather-7kk4f/must-gather-c4fs5" Oct 31 01:16:45.714505 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.714447 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h546x\" (UniqueName: \"kubernetes.io/projected/4cb51f18-afa0-4067-bb42-729af3290db1-kube-api-access-h546x\") pod \"must-gather-c4fs5\" (UID: \"4cb51f18-afa0-4067-bb42-729af3290db1\") " pod="openshift-must-gather-7kk4f/must-gather-c4fs5" Oct 31 01:16:45.752637 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.752580 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpdm\" (UniqueName: \"kubernetes.io/projected/29f5d09c-97b3-4ad7-af24-04464a0da0d1-kube-api-access-8lpdm\") pod \"must-gather-cjcfd\" (UID: \"29f5d09c-97b3-4ad7-af24-04464a0da0d1\") " pod="openshift-must-gather-m9n5w/must-gather-cjcfd" Oct 31 01:16:45.752760 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.752650 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29f5d09c-97b3-4ad7-af24-04464a0da0d1-must-gather-output\") pod \"must-gather-cjcfd\" (UID: \"29f5d09c-97b3-4ad7-af24-04464a0da0d1\") " pod="openshift-must-gather-m9n5w/must-gather-cjcfd" Oct 31 01:16:45.752887 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.752863 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7kk4f/must-gather-c4fs5" Oct 31 01:16:45.752981 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.752958 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29f5d09c-97b3-4ad7-af24-04464a0da0d1-must-gather-output\") pod \"must-gather-cjcfd\" (UID: \"29f5d09c-97b3-4ad7-af24-04464a0da0d1\") " pod="openshift-must-gather-m9n5w/must-gather-cjcfd" Oct 31 01:16:45.813538 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.813509 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpdm\" (UniqueName: \"kubernetes.io/projected/29f5d09c-97b3-4ad7-af24-04464a0da0d1-kube-api-access-8lpdm\") pod \"must-gather-cjcfd\" (UID: \"29f5d09c-97b3-4ad7-af24-04464a0da0d1\") " pod="openshift-must-gather-m9n5w/must-gather-cjcfd" Oct 31 01:16:45.863784 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:45.863756 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m9n5w/must-gather-cjcfd" Oct 31 01:16:46.131582 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:46.131552 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7kk4f/must-gather-c4fs5"] Oct 31 01:16:46.133720 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:16:46.133688 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cb51f18_afa0_4067_bb42_729af3290db1.slice/crio-97b5e9a6348ff14e1cb793f8d02f935bcb4318a56eb072662daeb2085f8349ea WatchSource:0}: Error finding container 97b5e9a6348ff14e1cb793f8d02f935bcb4318a56eb072662daeb2085f8349ea: Status 404 returned error can't find the container with id 97b5e9a6348ff14e1cb793f8d02f935bcb4318a56eb072662daeb2085f8349ea Oct 31 01:16:46.135369 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:46.135353 4969 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:16:46.241367 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:46.241339 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m9n5w/must-gather-cjcfd"] Oct 31 01:16:46.243238 ip-10-0-141-63 kubenswrapper[4969]: W1031 01:16:46.243212 4969 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29f5d09c_97b3_4ad7_af24_04464a0da0d1.slice/crio-6bf59ce0b1971bf6781df0a7516b7a4d2b85f5c064bc908979f55c34c47ab6e8 WatchSource:0}: Error finding container 6bf59ce0b1971bf6781df0a7516b7a4d2b85f5c064bc908979f55c34c47ab6e8: Status 404 returned error can't find the container with id 6bf59ce0b1971bf6781df0a7516b7a4d2b85f5c064bc908979f55c34c47ab6e8 Oct 31 01:16:46.862629 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:46.862576 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7kk4f/must-gather-c4fs5" event={"ID":"4cb51f18-afa0-4067-bb42-729af3290db1","Type":"ContainerStarted","Data":"97b5e9a6348ff14e1cb793f8d02f935bcb4318a56eb072662daeb2085f8349ea"} Oct 31 01:16:46.863885 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:46.863860 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m9n5w/must-gather-cjcfd" event={"ID":"29f5d09c-97b3-4ad7-af24-04464a0da0d1","Type":"ContainerStarted","Data":"6bf59ce0b1971bf6781df0a7516b7a4d2b85f5c064bc908979f55c34c47ab6e8"} Oct 31 01:16:47.873089 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:47.873032 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m9n5w/must-gather-cjcfd" event={"ID":"29f5d09c-97b3-4ad7-af24-04464a0da0d1","Type":"ContainerStarted","Data":"4b5a3359e20722bfc3ce048f5d6db31726be61c9e64b770dfec143fba635fda1"} Oct 31 01:16:47.873543 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:47.873108 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m9n5w/must-gather-cjcfd" event={"ID":"29f5d09c-97b3-4ad7-af24-04464a0da0d1","Type":"ContainerStarted","Data":"c66dcae4cabfb9f54efb9d1ce176aa2f6b3cdef039f4b8e3ea6b3c72b1bfd273"} Oct 31 01:16:47.989956 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:47.989886 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-m9n5w/must-gather-cjcfd" podStartSLOduration=1.799095351 podStartE2EDuration="2.989865623s" totalImagesPullingTime="1.190770272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:47.987869847 +0000 UTC m=+4440.463228270" watchObservedRunningTime="2026-10-31 01:16:47.989865623 +0000 UTC m=+4440.465224047" Oct 31 01:16:53.927421 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:53.927377 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7kk4f/must-gather-c4fs5" event={"ID":"4cb51f18-afa0-4067-bb42-729af3290db1","Type":"ContainerStarted","Data":"16107117ae5ee1f36eba91e8f1cebf60ebc39e8dfe5695f31558f009dc00810f"} Oct 31 01:16:53.928008 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:53.927977 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7kk4f/must-gather-c4fs5" event={"ID":"4cb51f18-afa0-4067-bb42-729af3290db1","Type":"ContainerStarted","Data":"008ced6011651407592cc0f9e2c44c675ef748fd854aad4ee9e7aaae8a1b276d"} Oct 31 01:16:54.043191 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:54.042571 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7kk4f/must-gather-c4fs5" podStartSLOduration=2.189227677 podStartE2EDuration="9.042552093s" totalImagesPullingTime="6.853324416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:54.041461736 +0000 UTC m=+4446.516820158" watchObservedRunningTime="2026-10-31 01:16:54.042552093 +0000 UTC m=+4446.517910517" Oct 31 01:16:54.856212 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:54.856176 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-24nkz_c7ae8cb8-c2ec-4476-bc1d-c13e3ae10454/global-pull-secret-syncer/0.log" Oct 31 01:16:56.730636 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:56.728912 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-czswt_69851eab-3b0f-4805-b6fe-2e58e42c21b0/connection-checker/0.log" Oct 31 01:16:57.169625 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:57.168249 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fzds7_143a4be9-3aa9-4329-88a7-71e995e72ca8/connection-checker/0.log" Oct 31 01:16:57.618940 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:57.618825 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rgl6b_584395de-91ba-4df6-84c3-0320c0d81cd6/connection-checker/0.log" Oct 31 01:16:58.392441 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:58.392357 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jgwjj_fb9cbf01-d844-41ce-8778-56c3d92f9862/server/0.log" Oct 31 01:16:58.539656 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:58.538345 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5hrcq_0fefd28c-b0d1-4863-819d-2d273ea5e9ef/konnectivity-agent/0.log" Oct 31 01:16:59.853404 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:16:59.850441 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-63.us-west-1.compute.internal_020b2f1cd0c673bc8a3307af2e07b1c3/haproxy/0.log" Oct 31 01:17:00.076004 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:00.075964 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-svrwj_4ce42331-4673-4ade-add7-cc6a77a308ec/proxy/0.log" Oct 31 01:17:01.000373 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:01.000346 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jgwjj_fb9cbf01-d844-41ce-8778-56c3d92f9862/server/0.log" Oct 31 01:17:02.764998 ip-10-0-141-63 kubenswrapper[4969]: E1031 01:17:02.764959 4969 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hzjx9" containerName="extractor" Oct 31 01:17:02.886943 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:02.886908 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-svrwj_4ce42331-4673-4ade-add7-cc6a77a308ec/proxy/0.log" Oct 31 01:17:05.054925 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.054898 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jgwjj_fb9cbf01-d844-41ce-8778-56c3d92f9862/server/0.log" Oct 31 01:17:05.150941 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.150905 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b5nfc_40085cb1-46da-478b-bf95-6da06fd5e85f/manager/0.log" Oct 31 01:17:05.574126 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.574087 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zsk4m_7e3ba2f7-5e86-496b-82fb-956ac8db9d60/manager/0.log" Oct 31 01:17:05.589999 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.589955 4969 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k"] Oct 31 01:17:05.596438 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.596408 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.683223 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.683184 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k"] Oct 31 01:17:05.695493 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.695461 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-podres\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.695493 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.695492 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-sys\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.695783 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.695600 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-lib-modules\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.695783 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.695634 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngmnk\" (UniqueName: \"kubernetes.io/projected/91a824ac-6e32-48b8-92cb-8455dabc9ff5-kube-api-access-ngmnk\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.695783 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.695722 4969 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-proc\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.796569 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.796533 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-lib-modules\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.796569 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.796567 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngmnk\" (UniqueName: \"kubernetes.io/projected/91a824ac-6e32-48b8-92cb-8455dabc9ff5-kube-api-access-ngmnk\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.796859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.796629 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-proc\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.796859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.796719 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-proc\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.796859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.796729 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-podres\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.796859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.796777 4969 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-sys\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.796859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.796782 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-podres\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.796859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.796741 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-lib-modules\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.796859 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.796822 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91a824ac-6e32-48b8-92cb-8455dabc9ff5-sys\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.857908 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.857834 4969 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngmnk\" (UniqueName: \"kubernetes.io/projected/91a824ac-6e32-48b8-92cb-8455dabc9ff5-kube-api-access-ngmnk\") pod \"perf-node-gather-daemonset-wrg2k\" (UID: \"91a824ac-6e32-48b8-92cb-8455dabc9ff5\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.916770 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.916742 4969 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:05.995176 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:05.995151 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dnvj9_64c1ce0f-d843-4306-8d6f-8c37a72ec455/kuadrant-console-plugin/0.log" Oct 31 01:17:06.301397 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:06.301371 4969 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k"] Oct 31 01:17:06.537507 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:06.537470 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-svrwj_4ce42331-4673-4ade-add7-cc6a77a308ec/proxy/0.log" Oct 31 01:17:07.069951 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:07.069901 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" event={"ID":"91a824ac-6e32-48b8-92cb-8455dabc9ff5","Type":"ContainerStarted","Data":"6b4ca723ed4ae1b70954ef0798daa157e0345c6de7486ac4e68d99be955f4901"} Oct 31 01:17:07.070178 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:07.070010 4969 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" event={"ID":"91a824ac-6e32-48b8-92cb-8455dabc9ff5","Type":"ContainerStarted","Data":"e9edd2bc58e012e1943674defe7ad8f0ce3aa90a22880d4d2da68f33b35e8a45"} Oct 31 01:17:07.070178 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:07.070067 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:07.187858 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:07.187529 4969 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" podStartSLOduration=2.187515503 podStartE2EDuration="2.187515503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:07.185808548 +0000 UTC m=+4459.661166972" watchObservedRunningTime="2026-10-31 01:17:07.187515503 +0000 UTC m=+4459.662873926" Oct 31 01:17:08.089500 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:08.089463 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jgwjj_fb9cbf01-d844-41ce-8778-56c3d92f9862/server/0.log" Oct 31 01:17:09.434724 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:09.434689 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-svrwj_4ce42331-4673-4ade-add7-cc6a77a308ec/proxy/0.log" Oct 31 01:17:11.437156 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:11.437128 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jgwjj_fb9cbf01-d844-41ce-8778-56c3d92f9862/server/0.log" Oct 31 01:17:12.888396 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:12.888366 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-svrwj_4ce42331-4673-4ade-add7-cc6a77a308ec/proxy/0.log" Oct 31 01:17:13.086149 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:13.086122 4969 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-wrg2k" Oct 31 01:17:13.517135 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:13.517094 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-sn7fw_fabccc0d-dbcd-4185-b0dd-1ba72182cc94/cluster-monitoring-operator/0.log" Oct 31 01:17:14.157041 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:14.157009 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jgwjj_fb9cbf01-d844-41ce-8778-56c3d92f9862/server/0.log" Oct 31 01:17:15.553245 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:15.553214 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-svrwj_4ce42331-4673-4ade-add7-cc6a77a308ec/proxy/0.log" Oct 31 01:17:16.488064 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:16.488035 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6_3dad313d-11a2-440b-a9f9-c0509d6bbcc6/istio-proxy/0.log" Oct 31 01:17:18.673169 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:18.673136 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94bcddf7f-vbt59_2b251574-895b-4ac8-9688-3e094921637a/router/0.log" Oct 31 01:17:18.710626 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:18.710580 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6rcr_27dea15a-9318-4398-a940-aa43ab973e44/node-exporter/0.log" Oct 31 01:17:19.092342 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:19.092257 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6_3dad313d-11a2-440b-a9f9-c0509d6bbcc6/istio-proxy/0.log" Oct 31 01:17:19.187478 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:19.187449 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6rcr_27dea15a-9318-4398-a940-aa43ab973e44/kube-rbac-proxy/0.log" Oct 31 01:17:19.607411 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:19.607380 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s6rcr_27dea15a-9318-4398-a940-aa43ab973e44/init-textfile/0.log" Oct 31 01:17:21.301394 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:21.301366 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-94bcddf7f-vbt59_2b251574-895b-4ac8-9688-3e094921637a/router/0.log" Oct 31 01:17:24.279188 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:24.279158 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b5nfc_40085cb1-46da-478b-bf95-6da06fd5e85f/manager/0.log" Oct 31 01:17:24.486399 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:24.486363 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zsk4m_7e3ba2f7-5e86-496b-82fb-956ac8db9d60/manager/0.log" Oct 31 01:17:24.695113 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:24.695088 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dnvj9_64c1ce0f-d843-4306-8d6f-8c37a72ec455/kuadrant-console-plugin/0.log" Oct 31 01:17:27.138170 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:27.138129 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b5nfc_40085cb1-46da-478b-bf95-6da06fd5e85f/manager/0.log" Oct 31 01:17:27.348303 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:27.348264 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zsk4m_7e3ba2f7-5e86-496b-82fb-956ac8db9d60/manager/0.log" Oct 31 01:17:27.563432 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:27.563353 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dnvj9_64c1ce0f-d843-4306-8d6f-8c37a72ec455/kuadrant-console-plugin/0.log" Oct 31 01:17:29.510353 ip-10-0-141-63 kubenswrapper[4969]: I1031 01:17:29.510321 4969 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c574b7b6f-b5mx6_3dad313d-11a2-440b-a9f9-c0509d6bbcc6/istio-proxy/0.log"